Activation of Ancillary Sensor Systems Based on Triggers from a Wearable Gesture Sensing Device

ABSTRACT

An event detection system includes sensors to detect movement and other physical inputs related to a user, which the event detection system can process to identify gestures of the user, and possibly also determine, using historical data, machine learning, rule sets, or other techniques for processing data to derive an inferred event related to the user sensed by the sensors. An inferred event might be an eating event, a smoking event, a personal hygiene event, a medication related event, or some other event the user is inferred to be engaging in. When an event is inferred to have started, to be ongoing, and/or to have concluded, the event detection system can take actions related to that event, such as obtaining other information to be stored in memory in association with the data representing the event, interacting with the user to provide information or reminders or to prompt for user input, sending a message to a remote computer system, sending a message to another person, such as a friend, health care provider, first responder, or other action(s).

CROSS-REFERENCES TO PRIORITY AND RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication 62/623,418, filed Jan. 29, 2018, entitled “Activation ofAncillary Sensor Systems Based on Triggers from a Wearable GestureSensing Device”.

This application also claims the benefit of U.S. Provisional PatentApplication 62/753,819, filed Oct. 31, 2018, entitled “AutomatedDetecting of a Triggering Event and Signaling to a Medication DispensingSystem or Messaging to a Patient or Caregiver”.

This application claims priority from the co-pending applications U.S.patent application Ser. No. 15/419,996 filed Jan. 30, 2017, entitled“Method and Apparatus for Tracking of Food Intake and Other Behaviorsand Providing Relevant Feedback” (“Vleugels I”) and (2) U.S. patentapplication Ser. No. 15/835,361 filed Dec. 7, 2017 entitled “Method andApparatus for Tracking of Food Intake and Other Behaviors and ProvidingRelevant Feedback” (“Vleugels II”). Vleugels I in turn claims thebenefit of U.S. Provisional Patent Application 62/288,408, filed Jan.28, 2016, entitled “Method and Apparatus for Food Intake Tracking andFeedback” and Vleugels II in turn claims the benefit of U.S. ProvisionalPatent Application 62/431,330, filed Dec. 7, 2016, entitled “MachineClassification of Temporally-Limited Gesture Inputs”.

Each of the above-cited applications or patents are hereby incorporatedby reference, as if set forth in full in this document, for allpurposes.

FIELD OF THE INVENTION

The present invention relates generally to systems of sensors thatmonitor users and determine user activities and more particularly tomethods and apparatus for tracking user activity including thetriggering of ancillary sensor systems when particular gestures orevents of the user are sensed.

BACKGROUND

Gesture-sensing technology can be used to automatically detect a gestureevent without user prompting or interaction, such as gestures toindicate events when someone is eating or drinking from their handgestures using motion sensors (accelerometer/gyroscope) in a wrist-wornwearable device or ring. This detection can occur in real time to deducekey insights about the consumption activity such as, for example, astart time of a consumption event, an end time of the consumption event,a consumption method, metrics associated with pace of consumption,metrics associated with quantities consumed, and metrics associated withfrequency of consumption, location of consumption, etc. Thegesture-sensing technology can be used for other activities andbehaviors such as smoking, dental hygiene, hand hygiene, etc. Examplesare described in Vleugels I and Vleugels II.

Tracking food and nutrition intake can be error-prone, incomplete,tedious and disruptive. Even when gestures are sensed, it might requireconsiderable user involvement and interruption to then manually processthe sensed information, such as by logging. Software installed on oraccessed from a tablet, mobile phone, laptop or computer, couldfacilitate logging and tracking of a person's food intake, but oftenthat requires user intervention.

Specialty devices, such as tableware and utensils with built-in sensorshave been proposed to track food intake more automatically. For example,a plate with integrated sensors and circuitry might automaticallyquantify and track the content of food that is placed on the plate.Similarly, integrated sensors in a drinking vessel might identify,quantify and track the contents of liquid in the cup. In anotherexample, an eating utensil includes sensors that count the number ofbites taken by a person using the eating utensil. These methods mightfall short in not being able to automatically identify and quantify thecontent of the food being consumed and also only apply to a limited setof meal scenarios and dining settings and are not well suited toproperly cover the wide range of different meal scenarios and diningsettings that a typical person may encounter during a day. Additionally,these are not typically configured to integrate with electronic datafrom external sensing systems.

Being able to handle a wide variety of meal scenarios and settings isimportant for seamless and comprehensive food intake tracking. A methodbased on an eating utensil may not be able to properly track the intakeof drinks, snacks or finger foods and such methods may also interferewith a person's normal social behavior. For example, it might not besocially acceptable for a user to bring their own eating utensils to arestaurant or a dinner at a friend's house.

Devices and methods have been described that quantify and track foodintake based on analysis of images of food taken by a portable devicethat has imaging capabilities, such as an app that runs on a mobilephone or tablet that has a camera. Some devices might use spectroscopyto identify food items based on their molecular makeup. Such devices mayuse crowd sourcing and/or computer vision techniques, sometimescomplemented with other image processing techniques, to identify a fooditem, estimate its nutritional content and/or estimate its portion size.However, many of these devices and methods are fond lacking in usabilityand availability in certain social settings.

Improved methods and apparatus for advanced gesture and event monitoringand analysis are needed.

SUMMARY

Using gesture sensing technology, an event detection system can triggeran ancillary device to gather further information or control externalobjects or devices. The ancillary device might operate independently, orwithout particular awareness of the nature of the event detectionsystem. The ancillary device might be related to a focus of the eventdetection system, or might be unrelated in function, or at leastapparently unrelated in function. For example, the event detectionsystem might be a food/drinking event detection system and the ancillarydevice might be a lighting controller, wherein the food/drinking eventdetection system detects the possible start of an undesirableeating/drinking event and sends a signal to an ancillary device that isthe lighting controller, triggering it to alter lighting in a room tochange the tone/intensity of the light in the room to positivelyinfluence behaviors.

In a specific embodiment, the external device is a NFC reader andvarious objects having NFC tags thereon are detected. Where thoseobjects are food/beverage related, the event detection system candetermine what the gestures are related to. For example, food/beveragecontainers might have NFC tags embedded in the product packaging and afood intake monitoring system might automatically determine thatgestures are related to an eating event, then signal to an NFC reader toturn on and read nearby NFC tags, thereby reading the NFC tags on theproducts being consumed so that the gestures and the event areassociated with a specific product.

In other variations, other wireless technology can be used. In somevariations, the external device is a module integrated within a housingthat also houses the event detection system. In some variations, tagsare read, such as NFC tags or battery-free Bluetooth tags. These tagsmight be powered by energy emitted from a reader or powered in otherways.

An event detection system might include sensors to detect movement andother physical inputs related to a user, which the event detectionsystem can process to identify gestures of the user, and possibly alsodetermine, using historical data, machine learning, rule sets, or othertechniques for processing data to derive an inferred event related tothe user sensed by the sensors. For example, sensors might detect audiosignals near the mouth of the user, which the event detection system canuse in inferring an event, such as an event related to a user's eatingor drinking activities. Other nonhand sensors might also be used, suchas motion sensors, temperature sensors, audio sensors, heart ratesensors, blood pressure sensors, etc.

Example gestures might be a food intake gesture, a sip gesture, or someother gesture. An inferred event might be an eating event, a smokingevent, a personal hygiene event, a medication related event, or someother event the user is inferred to be engaging in. A gesture mightrepresent some movement of the user, such as hand gestures that can bedetected using a wrist-worn device so as to be non-intrusive orminimally intrusive and that can operate with no or minimal userintervention. With an inferred event, the event detection system wouldproceed as if they event occurred, or is occurring, even if in fact thatevent is not occurring.

When an event is inferred to have started, to be ongoing, and/or to haveconcluded, the event detection system can take actions related to thatevent, such as obtaining other information to be stored in memory inassociation with the data representing the event, interacting with theuser to provide information or reminders, to coach the user or to promptfor user input, sending a message to a remote computer system, sending amessage to another person, such as a friend, health care provider, firstresponder, or other action(s). In a specific example, once the eventdetection system infers that an event has started, it signals to anancillary sensor system or an ancillary processing system to take anaction such as gathering more information, sending communication or aprocessing task. The event detection system might create a data recordfor an event once a new event is detected and populate that data recordwith details of the event that the event detection system is able todetermine, such as details about gestures that are involved in thatevent. The ancillary sensor system or ancillary processing system mightpopulate that data record with ancillary data about the event, such asthe ancillary data described herein.

In other variants, the ancillary sensor system or ancillary processingsystem may communicate ancillary data about the event to the eventdetection system and the event detection system may populate that datarecord with the ancillary data about the event. Other variants of howthe data record associated with an event gets populated are alsopossible.

The event detection system and ancillary sensor system and/or ancillaryprocessing system can be used as part of a monitoring system with one ormore applications, such as food logging, inventory tracking orreplenishment, production line monitoring, QC automation, medicationadherence, insulin therapy, supporting an automated insulin deliverysystem (also known as an artificial pancreas), and other applications.

The event detection system might be implemented with hardware and/orsoftware. The signals sent to the ancillary system can be signalsaccording to a protocol expected by the ancillary system that cause theancillary system to react in expected ways and not necessarily be awarethat it is an event detection system that is sending the signal. In thismanner, ancillary systems that are not even designed to work with anevent detection system will perform their operations upon beingtriggered by the event detection system.

An event detection system can include sensors to detect movement and/orother physical inputs related to a user, which the event detectionsystem can process to identify gestures of the user, a method of furtherprocessing comprising: determining a type of an inferred event, whereinthe inferred event is an actual event occurring with or about the useror is, based on sensor inputs, deemed by the event detection system tobe an event; determining an external trigger time, based on one or moreof data related to the inferred event, the type of the inferred event,and/or a timing of the inferred event, wherein the external trigger timeis a time determined relative to an anchor time of the inferred event;and according to the external trigger time, initiating a computer-basedaction in response to the inferred event.

The sensors can include sensors external to a main event detection unit.The computer-based action in response to the inferred event can comprisesending a trigger signal to an ancillary system that is operableindependent of the event detection system and independent of the triggersignal. The event detection system can be configured to determine, usinghistorical data, machine learning, rule sets, and/or other techniquesfor processing data, an inferred event related to the user sensed by thesensors. The type of the inferred event can be at least one of a foodintake event, a drinking event, a smoking event, a personal hygieneevent, and/or a medication related event. The external trigger time canbe determined from when the inferred event is inferred to have started,when the inferred event is inferred to be ongoing, and/or when theinferred event is inferred to have concluded.

The computer-based action in response to the inferred event might be oneor more of (1) obtaining other information to be stored in memory inassociation with data representing the inferred event, (2) interactingwith the user to provide information, coaching advice or a reminder, (3)interacting with the user to prompt for user input, (4) sending amessage to a remote computer system, (5) sending one or more inputs to amedication dispensing system, and/or (6) sending a message to anotherperson.

The computer-based action can be inputs to the medication dispensingsystem and wherein the medication dispensing system is an automated, orpartially automated, insulin delivery system, such as an insulin dosagecalculator that computes information from an object informationretrieval subsystem about what is being consumed.

The methods might include updating one or more of an inventory database,a medication log, an inventory, and/or a production line database inresponse to the inferred event, retrieving information about at leastone object the user is interacting with, perhaps over a wireless link,such as one that uses one of (1) near-field-communication technology,(2) Bluetooth technology, (3) Bluetooth Low Energy technology, (4) aderivative of Bluetooth technology that is at least partially consistentwith Bluetooth technology, or (5) a derivative of Bluetooth Low Energytechnology that is at least partially consistent with Bluetooth LowEnergy technology.

A method of managing a food log based in part on sensor data fromsensors on a device worn by a user for logging the user's food intake isalso provided, that might include identifying a current eating event;further identifying one or more characteristics of the current event,from the current eating event and the characteristics of the currentevent, and from historical data of past eating events, using a trainingsystem to identify additional characteristics of a current eating event;and periodically uploading data representing characteristics of eatingevents to a food log server.

In another variation, a method might comprise managing a food log basedin part on sensor data from sensors on a device worn by a user forlogging the user's food intake, and also might include identifying acurrent eating event; further identifying one or more characteristics ofthe current event, and periodically uploading data representingcharacteristics of eating events to a food log server, comparinguploaded characteristics of food intake events to second set of eatingcharacteristics that have been obtained through a different mechanismand, as a result of comparing, take one or more of the following actions(1) merge characteristics from both datasets correspond to the same foodintake event into a single eating event data record of a food loggingsystem, and (2) adding entries to the food log for the events absentfrom the food log but represented by identified characteristics of thecurrent eating event.

Periodically uploading data might comprise periodically uploading dataat conclusions of detected eating events or at times that correspond toanchor times of detected eating events.

The data representing characteristics of eating events might be derivedfrom inferences derived from sensed user movements during eating events.The data representing characteristics of eating events might bepartially inferred from sensed user movements during eating events andmeal signatures are added to the food log so as to allow the user toedit the food log while being reminded of the meal signature.

Recording the inferred event in the food log might comprise recordingone or more of information inferred by the event detection system andinformation retrieved from one or more objects the user interacted withas part of the inferred event, wherein user interaction is detectedusing sensors on a device worn by a user and sensor signals from thosesensors or from devices designed to read electronic tags on the user orthe food. The food log might comprise information about eating ordrinking events including one or more of a time of an event, a durationof the event, a location of the event, metrics related to pace ofconsumption during the event, metrics related to quantities consumedduring the event, eating methods, and/or utensils used, and/orinformation about items being eaten or drunk, wherein such informationis obtained from electronically readable tags attached to the items, themethod further comprising sending a tag reader trigger signal to a tagreader to read the electronically readable tags when an inferred eventis detected. The user might be prompted to reposition the items and thetag reader to a relative position in which the tag reader can read theelectronically readable tags attached to the items.

In a method of sending signals or messages in response to an event timedrelative to the event, the method might include identifying behaviorindicators, from the behavior indicators and timestamps of the behaviorindicators, and from historical data of past behavior events andbehavior indicators, using a training system to predict likely futureevents including a time of such likely future events; and reading a ruleset; when the rule set indicates that a particular signal should be sentwhen a particular event is anticipated and when the particular event isindeed anticipated with at least a predefined confidence level,outputting the particular signal to an ancillary system; and when therule set indicates that a particular message should be sent when theparticular event is anticipated and when the particular event is indeedanticipated with at least the predefined confidence level, sending theparticular message.

Identifying behavior indicators might comprise detecting at least onegesture of a user wearing a wearable device having a plurality ofsensors to detect movement and other physical inputs related to a user,receiving sensor inputs from the plurality of sensors, reading data fromexternal data sources, and from the sensors inputs and the data,identifying the behavior indicators.

The particular event might be an eating event, wherein the particularsignal is a signal to trigger an insulin microdosing system and theparticular message is a message to the user of a wearable deviceindicating that the start of the eating event has been detected. Theparticular event might be such that it could not be predicted from thesensor inputs alone or the data from external sources alone. Theparticular event might be an eating event and behavior indicatorscorrelate with triggers, wherein triggers include one or more of a sleeppattern, a stress level, an activity level, the user's location, peoplesurrounding the person, vital signs, a hydration level, a fatigue level,or a heart rate. The signal or message might be a function of a level ofuser engagement, and further comprise assessing the level of userengagement at a given time or over a time window, combined with userresponses to certain messages.

The following detailed description together with the accompanyingdrawings will provide a better understanding of the nature andadvantages of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an event monitoring system.

FIG. 2 illustrates a process to provide for user intervention.

FIG. 3 is an illustrative example of an environment in accordance withat least one embodiment.

FIG. 4 illustrates a behavior event detection or prediction system thatuses user historical data to determine when to send a signal or amessage that relates to detected or predicted behavior events.

FIG. 5 illustrates some of the components disposed in an electronicdevice, in accordance with one embodiment.

FIG. 6 illustrates elements of a wearable bracelet or wristband.

FIG. 7 shows an example of a machine classification system that includesa cross-correlated analytics sub-system.

FIG. 8 shows a high level functional diagram of a medication dispensingsystem.

FIG. 9 is an illustrative example of a machine learning system thatmight be used with other elements described in this disclosure.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Various examples are provided herein of devices that a person would useto monitor, track, analyze and provide feedback on food intake, theintake process and timing and other relevant aspects of a person'seating, drinking and other consumption for various ends, such asproviding diet information and feedback. The data related to food intakeprocess might include, timing of the eating process, pace of eating,time since last food intake event, what is eaten, estimates of thecontents of what is eaten, etc.

While many of the examples described herein are related to food intakeevents, the methods and devices described herein are also applicable toother behavior events such as brushing teeth, flossing, smoking, bitingnails, taking medication, hand washing, putting on lipstick or mascara,shaving, making coffee, cooking, urinating, using the bathroom, driving,exercising or engaging in a specific sports activity. Other examples ofan event that may be detected by an event detection subsystem couldinclude an operator on a production line or elsewhere performing aspecific task or executing a specific procedure. Yet another examplecould be a robot or robotic arm performing a specific task or executinga specific procedure on a production arm or elsewhere.

In some instances, it can be useful to also track information aboutobjects or other subjects with which the subject interacts as theyengage in an activity or exhibits certain behaviors. As an example, whena user is eating, it may be useful to not only detect and track that auser is eating, but to also detect and/or track information about whatthe user is eating. For example, it can be useful to know the brand,nutritional content or other information about the food being consumed.Likewise, when a user is drinking, it may also be useful to know thebrand and content of the drink being consumed. When a user is brushinghis teeth, it may be useful to know the brand of the toothpaste, or whena user is smoking it may be useful to also know the brand, type ofproduct being lightened or nicotine content. When a user takes a pill,it may be useful to know what medication the user is taking and thedosage. It may not always be possible to derive this level of detailabout the objects from analyzing just the hand gestures of a user or bysimply processing the signals that are used to determine the onset oroccurrence of an event or behavior, but with the additional informationprovided by the ancillary sensor systems, this information can beobtained and used or stored with records of the inferred events.

Data can be obtained from some stationary device having sensors andelectronics, some mobile device having sensors and electronics that iseasily moved and carried around by a person, and/or from wearabledevices having sensors and electronics that a person attaches to theirperson or clothing, or is part of the person's clothing. In general,herein such devices are referred to as sensing devices. The data mightbe raw sensor data provided by the sensors capable of outputting data,or the data might be processed, sampled, or organized in some way so asto be data derived from the outputs of sensors.

Herein, the person having such a device and who's consumption is beingmonitored is referred to as the user but it should be understood thatthe device might be used unchanged in situations where the personconsuming, the person monitoring, and the person evaluating feedbackneed not all be the same person. Herein, what is consumed is referred toas food intake, but it should be clear that these devices can be used tomore generally track consumption and consumption patterns. A behaviortracking/feedback system as described herein might comprise one or morewearable devices and might also comprise one or more additional devicesthat are not worn. These additional devices might be carried by thewearer or kept nearby so that they can communicate with the wearabledevices. The behavior tracking/feedback system might also compriseremote elements, such as a remote cloud computing element and/or remotestorage for user information.

A wearable device might be worn at different locations on the wearer'sbody (i.e., the person monitoring their behavior) and the wearabledevice might be programmed or configured to account for thosedifferences, as well as differences from wearer to wearer. For example,a right-handed person may wear the device around his right wrist whereasa left-handed person may wear the device around his left wrist. Usersmay also have different preferences for orientation. For example, someusers may want the control buttons on one side, whereas other users mayprefer the control buttons on the opposite side. In one embodiment, theuser may manually enter the wrist preference and/or device orientation.

In some embodiments, sensing devices can sense, without requiring userinteraction, the start/end of a food intake event, the pace of eating,the pace of drinking, the number of bites, the number of sips, theestimation of fluid intake, and/or estimation of portion sizing.Operating with less human intervention, no human intervention, or onlyintervention not apparent to others will allow the devices to scale wellwith different meal scenarios and different social situations. Sensingmight include capturing details of the food before it is consumed, aswell as user actions that are known to accompany eating, such asrepeated rotation of an upper arm or other hand-to-mouth motions.Sensors might include an accelerometer, a gyroscope, a camera, and othersensors.

Using the devices can provide a person with low friction-of-use todetect, quantify, track and provide feedback related to the person'sfood intake content as well as the person's food intake behavior. Suchmethods have the potential of preventing, treating and, in certaincases, even curing diet-related diseases. Such devices can improveefficacy, accuracy and compliance, and reduce the burden of usage and toimprove social acceptance. The devices can operate autonomously with no,or very minimal, human intervention, and do not interfere in an invasiveor otherwise significant negative way with a person's normal activitiesor social interactions or intrude on the person's privacy. The devicesare able to handle a wide range of meal scenarios and dining settings ina discreet and socially-acceptable manner, and are capable of estimatingand tracking food intake content and quantity as well as other aspectsof eating behavior. The devices can provide both real-time andnon-real-time feedback to the person about their eating behavior, habitsand patterns.

The devices might be useful for a person concerned about their diet. Forexample, people with Type 1 diabetes are usually on an insulin therapywhere, based on their food intake and other factors, they administer theproper insulin dosage. While the cause of Type 1 diabetes may not bedirectly linked to a person's eating behavior, a person with Type 1diabetes needs to carefully track his or her food intake in order tomanage his or her insulin therapy. Such patients will also benefit fromeasier to use and more discreet methods for food intake tracking. Insome embodiments of the sensing devices, the sensing device is part of afeedback-driven automated insulin delivery therapy system. Such a systemmight include one or more of continuous monitoring of a patient'sglucose levels, a precision insulin delivery system, and the use ofinsulin that has a faster absorption rate, that would further benefitfrom information that can be extracted from automated and seamless foodintake tracking, such as the tracking of carbohydrates and sugar intake.The devices might also be useful for wellness programs and the like.

A food intake event generally relates to a situation, circumstance oraction whereby a person eats, drinks or otherwise takes into his or herbody an edible substance. Edible substances may include, but are notlimited to, solid foods, liquids, soups, drinks, snacks, medications,vitamins, drugs, herbal supplements, finger foods, prepared foods, rawfoods, meals, appetizers, main entrees, desserts, candy, breakfast,sports or energy drinks. Edible substances include, but are not limitedto, substances that may contain toxins, allergens, viruses, bacteria orother components that may be harmful to the person, or harmful to apopulation or a subset of a population. Herein, for readability, food isused as an example of an edible substance, but it should be understoodthat other edible substance might be used instead of food unlessotherwise indicated.

Eating habits and patterns generally relate to how people consume food.Eating habits and patterns may include, but are not limited to, the paceof eating or drinking, the size of bites, the amount of chewing prior toswallowing, the speed of chewing, the frequency of food intake events,the amount of food consumed during a food intake event, the position ofthe body during a food intake event, possible movements of the body orof specific body parts during the food intake event, the state of themind or body during a food intake event, and the utensils or otherdevices used to present, handle or consume the food. The pace of eatingor drinking might be reflected in the time between subsequent bites orsips.

Triggers generally relate to the reasons behind the occurrence of a foodintake event, behind the amount consumed and behind how it is consumed.Triggers for food intake events and for eating habits or patterns mayinclude, but are not limited to, hunger, stress, social pressure,fatigue, addiction, discomfort, medical need, physical location, socialcontext or circumstances, odors, memories or physical activity. Atrigger may coincide with the food intake event for which it is atrigger. Alternatively, a trigger may occur outside the food intakeevent window, and might occur prior to or after the food intake event ata time that may or may not be directly related to the time of the foodintake event.

In some embodiments of the sensing device or system, fewer than all ofthe features and functionality presented in this disclosure areimplemented. For example, some embodiments may focus solely on detectionand/or processing and tracking of the intake of food without intendingto steer the user to modify his or her food intake or without tracking,processing or steering eating habits or patterns.

In many examples herein, the setting is that an electronic device isprovided to a user, who wears the electronic device, alone or while itis in communication with a nearby support device that might or might notbe worn, such as a smartphone for performing operations that the wornelectronic device offloads. In such examples, there is a person wearingthe electronic device and that person is referred to as the “wearer” inthe examples and the system comprises a worn device and may includeother components that are not worn and are nearby and components thatare remote, preferably able to communicate with the worn device. Thus,the wearer wears the electronic device, and the electronic deviceincludes sensors, which sense environment about the wearer. That sensingcan be of ambient characteristics, body characteristics, movement andother sensed signals as described elsewhere herein.

In many examples, functionality of the electronic device might beimplemented by hardware circuitry, or by program instructions that areexecuted by a processor in the electronic device, or a combination.Where it is indicated that a processor does something, it may be thatthe processor does that thing as a consequence of executing instructionsread from an instruction memory wherein the instructions provide forperforming that thing. While other people might be involved, a commonexample here is where the wearer of the electronic device is using thatelectronic device to monitor their own actions, such as gestures,behavior events comprising a sequence of gestures, activities, starts ofactivities or behavior events, stops of activities or behavior events,etc. Where it is described that a processor performs a particularprocess, it may be that part of that process is done separate from theworn electronic device, in a distributed processing fashion. Thus, adescription of a process performed by a processor of the electronicdevice need not be limited to a processor within the worn electronicdevice, but perhaps a processor in a support device that is incommunication with the worn electronic device.

FIG. 1 shows a high level functional diagram of a monitoring system of asubject in accordance with an embodiment of the present invention. Asshown in FIG. 1, sensor units 100 interact with an event detectionsystem 101, that in turn interacts with an object information retrievalsystem 102, that provide inputs to a processing and analysis system 103,the results of which can be stored in data storage units 104.

Other variants are also possible. For example, object informationretrieval system 102 may provide inputs back to event detection system101 that in turn forward inputs to processing and analysis system 103.Other variants of how the data gets processed and stored in data storageunit 104 are also possible.

In some embodiments, elements shown in FIG. 1 are implemented inelectronic hardware, while in others some elements are implemented insoftware and executed by a processor. Some functions might sharehardware and processor/memory resources and some functions might bedistributed. Functionality might be fully implemented in a sensor devicesuch as wrist worn wearable device, or functionality might beimplemented across the sensor device, a processing system that thesensor device communicates with, such as a smartphone, and/or a serversystem that handles some functionality remote from the sensor device.For example, a wearable sensor device might make measurements andcommunicate them to a mobile device, which may process the data receivedfrom the wearable sensor device and use that information possiblycombined with other data inputs, to activate object informationretrieval subsystem 102. Object information retrieval subsystem 102 maybe implemented on the mobile device, on the wearable sensor device, oron another electronic device. The object information retrieval subsystem102 may also be distributed across multiple devices such as for exampleacross the mobile device and the wearable sensor device. Data or otherinformation may be stored in a suitable format, distributed overmultiple locations or centrally stored, in the form recorded, or aftersome level of processing. Data may be stored temporarily or permanently.Data may be stored locally on the wearable sensor device, on the mobiledevice or may be uploaded over the Internet to a server.

Monitoring System Overview

A first component of the system illustrated in FIG. 1 is the eventdetection subsystem 101. One role of event detection subsystem 101 is toidentify the actual, probable or imminent occurrence of an event. Anevent could, for example, be an event related to a specific activity orbehavior. Specific activities or behaviors may include, but are notlimited to, eating, drinking, smoking, taking medication, brushingteeth, flossing, hand washing, putting on lipstick or mascara, shaving,making coffee, cooking, urinating, using the bathroom, driving,exercising or engaging in a specific sports activity. Other examples ofan event that may be detected by event detection subsystem 101 could bean operator on a production line or elsewhere performing a specific taskor executing a specific procedure. Yet another example could be a robotor robotic arm performing a specific task or executing a specificprocedure on a production arm or elsewhere.

The event detection subsystem may use inputs from one or more sensorunits 100, other user inputs 105, or a combination of one or more sensorinputs from sensor unit 100 and one or more other user inputs 105 todetermine or infer the actual, probable or imminent occurrence of anevent. The event detection subsystem 101 may do additional processing onthe sensor and/or user inputs to determine the actual, probable orimminent occurrence of an event. In general, the event detection systemrecords and/or reacts to an inferred event, which occurs when the eventdetection system has inputs and/or data from which it determines that anevent might actually be starting, might likely start, or is imminent. Insome cases, the event detection system might infer an event where anevent did not actually occur and process it as an event, but this mightbe an infrequent occurrence.

The event detection subsystem 101 may also do additional processing onthe sensor and/or user inputs to determine additional information aboutthe event. Such information may include, but is not limited to, theduration of the event, the event start time, the event end time, metricsassociated with the speed or pace at which subject is engaging in theevent. Other event data elements are also possible. For example, if theevent is an eating event, an event data element could be the number ofbites or the amount of food consumed. Similarly, if the event is adrinking event, an event data element could be the number of sips or theamount of fluid intake. These event data elements might be stored in adatabase that maintains data elements about inferred events.

Using the gesture sensing technology, an event detection system cantrigger and external device to gather further information.

In a specific embodiment, the electronic device that houses objectinformation retrieval subsystem 102 or a portion of object informationretrieval subsystem 102 includes Near-Field-Communication (NFC)technology and the object information retrieval subsystem 102 obtainsinformation about the object(s) or subject(s) with which the subject maybe interacting at least in part via transmission over a wireless NFClink.

In a specific embodiment, the external device is a NFC reader andvarious objects having NFC tags thereon are detected. The NFC readermight be integrated with the gesture sensing technology or with somecomponents of the gesture sensing technology.

Where those objects are food/beverage related, the event detectionsystem can determine what the gestures are related to. For example,food/beverage containers might have NFC tags embedded in the productpackaging and a food intake monitoring system might automaticallydetermine that gestures are related to an eating event, then signal toan NFC reader to turn on and read nearby NFC tags, thereby reading theNFC tags on the products being consumed so that the gestures and theevent are associated with a specific product.

In an example, a monitoring system might have a wearable device thatdetermines gestures and based on those gestures, identifies an eatingevent or a drinking event. Suppose a drinking event is detected andbased on the sensor inputs and gestures detected, the monitoring systemdetermined that the user drank three quarters of a can of soda, such asbe counting sips and estimating or computing the size of each sip. Asthe gestures are likely identical regardless of whether it is a dietsoda or a regular soda. Using the NFC reader, the specific brand andtype of soda can be detected as well.

The sensors may be in the wearable device, with gesture determinationlogic or processing occurring in an external device that iscommunicatively coupled to the wearable device, such as a mobile phone,or the gesture determination may partially happen on the wearable deviceand partially on the external device that is communicatively coupled tothe wearable device.

The NFC reader may be in the wearable device that houses the sensors, inan external device that is communicatively coupled to the wearabledevice and that performs at least some of the gesture determination orin another external device that is communicatively coupled to thewearable device, to an external device that performs at least some ofthe gesture determination or to both.

In the general case, the detection of the occurrence of an event may beused to initiate a process/system/circuitry/device to collectinformation about objects/items or other subjects that are beinginteracted with by the person performing the activity or behaviorrepresented by the event. This information may be recorded in the formof data elements. Object data elements may be stored in a database. Oneor more object data elements and one or more event data elements of thesame event may be recorded as a single entry in a database. Object dataelements and event data elements may also be recorded as separateentries in a database. Other data structures consistent with theteachings herein might also be used, or used instead.

When the NFC reader system gets activated, it initiates one or more NFCread commands, and additional information from relevant objects isobtained wirelessly over NFC link. Additional processing may be applied,such as filtering out the NFC data to simplify processing downstream.

In other variations, other wireless links are used instead of NFC orwith NFC. Examples of other wireless technologies include but are notlimited to Bluetooth, Bluetooth Low Energy, Wi-Fi, Wi-Fi derivatives,and proprietary wireless technologies.

The wireless module or device used by object information retrievalsubsystem 102 may also be used for other purposes. For example, if thewireless module is a Bluetooth or Bluetooth Low Energy module, and theobject information retrieval subsystem is housed inside a smartphone,the wireless module may also be used for wireless communication withother devices that are wirelessly connected to the smartphone, such asfor example wireless earphones, a wireless continuous glucose monitoringdevice, a wireless entertainment system.

The wireless module may use a certain wireless communication protocol(“primary protocol”), such as Bluetooth or Bluetooth Low Energy tocommunicate with the other devices. When used by object retrievalinformation subsystem 102 to retrieve information from objects thewearer may be interacting with, the wireless module may use a modifiedwireless communication protocol (“secondary protocol”) for example, toreduce power consumption of the objects it is retrieving informationfrom. The secondary protocol may be partially, but not fully, compliantwith the primary protocol. For example, the secondary protocol may onlyimplement a subset of the functionality of the primary protocol, mayhave different limits on maximum packet size, may have longer timeintervals between packet exchanges, may support only a subset of thesupported data rates, may use modified frame formats. In general, apartially compliant secondary protocol is one where a device using thesecondary protocol can communicate with other devices using thesecondary protocol, and perhaps the primary protocol, where nearbydevices that are using the primary protocol are able to process at leastsomething about the secondary protocol communications, instead ofperhaps treating them as noise or interference, but in some way thedevice using the secondary protocol is not fully compliant with theprimary protocol. An example is low-power, or unpowered, Bluetooth tagsthat are able to communicate using parts of the Bluetooth protocol, butdo not implement all of the required or expected features of theBluetooth protocol.

As an example, a very low power tag might be powered solely by impingingelectrical fields (like an NFC tag) and might be a tag that implementsparts of the Bluetooth protocol, but not the more power-intensive parts,and can still communicate with a tag reader. It may be that an eventdetection system is agnostic to the type of tag reader, or it mightspecifically expect the tag reader to look for, and read, very low powertags. A wireless network protocol implemented by Bluetooth tag maysupport a subset of the protocol of the wireless module in the objectretrieval information subsystem. This would allow the wireless module inthe object retrieval information subsystem to operate as a regularBluetooth/Bluetooth LE device, but it can also receive/decode framessent by a lower-powered Bluetooth tag using a modified protocol withenough functionality implemented such that it can receive/decode packetsreceived from the main wireless module. Thus, in a specific embodiment,the external device is a Bluetooth module or a Bluetooth Low Energymodule and various objects having Bluetooth tags use energy scavengingas their sole or partial source of power and those tags can be detected.The Bluetooth module might be integrated with the gesture sensingtechnology or with some components of the gesture sensing technology.Other wireless network protocols are also possible.

The detection of occurrence of an event signal may be used to filter outinformation about specific, relevant objects that are related to theactivity/behavior of interest that are part of the event.

While the object detection process might be automatic, it can also haveuser intervention required to activate the object information retrievalsystem. For example, it could be as simple as asking the user to turn ona camera, or make a decision whether to continue the detection processand get the user's input on the decision. As another example, the usermay be prompted to move the NFC reader closer to the NFC tag ofinterest. In another example, the user may be prompted to activate theNFC reader circuitry, or portions of the NFC reader circuitry or takeone or more actions that allow the NFC reader circuitry to issue readcommand.

In addition to an NFC reader, a camera might be activated and additionalinformation from relevant objects is obtainable by analyzing images orvideo recording of relevant objects. The camera might be combined in aunit with the NFC reader. In some embodiments, only a camera is used, orother ancillary sensors are used to obtain additional information aboutthe event.

Information about the activity or behavior that are part of the event,obtained from the event detection subsystem can be combined withinformation about the object(s) or subject(s) the user is interactingwith, and additional processing/analysis may be performed on thecombined dataset to obtain additional information or insights about theactivity/behavior that cannot be obtained from looking at only one ofthe data sources alone.

While many examples in this disclosure refer to the event detectionsystem analyzing gestures to detect the actual, likely or imminentoccurrence of an event, other sensor inputs are also possible. Forexample, audio signals in or near the mouth, throat or chest may be usedto detect and obtain information about a user's consumption. Toaccomplish this, the event detection subsystem 101 may use inputs 107from one or more sensor units 100. Sensors may include, but are notlimited to, accelerometers, gyroscopes, magnetometers, magnetic angularrate and gravity (MARG) sensors, image sensors, cameras, opticalsensors, proximity sensors, pressure sensors, odor sensors, gas sensors,glucose sensors, heart rate sensors, ECG sensors, blood pressuresensors, thermometers, light meters, Global Positioning Systems (GPS),and microphones.

Methods for event detection may include, but are not limited to,detection based on monitoring of movement or position of the body or ofspecific parts of the body, monitoring of arm movement, position orgestures, monitoring of hand movement, position or gestures, monitoringof finger movement, position or gestures, monitoring of swallowingpatterns, monitoring of mouth and lips movement, monitoring of saliva,monitoring of movement of cheeks or jaws, monitoring of biting or teethgrinding, monitoring of signals from the mouth, the throat and thedigestive system. Methods for detection may include visual, audio or anyother types of sensory monitoring of the person or robot/machine and/orhis or her surroundings. Data analytics techniques, including but notlimited to statistics, machine learning and artificial intelligencetechniques may be applied to detect the actual, probably or imminentoccurrence of an event.

Upon the detection of an actual, probable or imminent occurrence of anevent, the event detection subsystem may activate the object informationretrieval subsystem 102. Alternatively, the event detection subsystem101 may do additional processing to decide whether or not to activatethe object information retrieval subsystem 102. The event detectionsubsystem 101 may activate the object information retrieval subsystem102 immediately or wait for a certain time before activating the objectinformation retrieval subsystem 102.

The role of object information retrieval subsystem 102 is to collectinformation about objects or other subjects a subject is interactingwith when or some time after the event detection subsystem 101 hasdetected the occurrence of an actual, probable or imminent event.

In one such embodiment, upon receiving an activation input signal 107from event detection subsystem 101, or some time after receiving anactivation input signal 107, object information retrieval subsystem 102initiates an NFC read action to read the NFC tag attached to, housedwithin or otherwise associated with one or more objects that are withinNFC range of the device that houses object information retrievalsubsystem 102. Object information retrieval subsystem 102 sends the datareceived from one or more objects to processing and analysis subsystem103. Object information retrieval subsystem 102 may do additionalprocessing on the data before sending it to processing and analysissubsystem 103. In other embodiments, additional ancillary sensors orsensor systems might be used, such as a camera or other electronicdevice.

Processing may include but is not limited to filtering, extractingspecific data elements, modifying data or data elements, combining dataor data elements obtained from multiple objects, combing data or dataelements with data obtained from other sources not collected via NFC.Examples of filtering may include but are not limited to filtering basedon distance or estimated distance between the object or objects andobject information retrieval subsystem, based on signal strength ofreceived NFC signals, based on order in which data is received fromobjects, based on information in the data or in specific data elements.Other filtering mechanisms or criteria used for filtering are alsopossible. Object information retrieval subsystem 102 may stop readingNFC tags after a fixed time, after a configurable time, after reading afixed or configurable number of tags. Other criteria may also be used.

It is also possible, that the object information retrieval subsystemcollects information about objects or other subjects a subject isinteracting with independent of the event detection subsystem. In suchan embodiment, the processing and analysis subsystem 103 may use signals108 received from event detection subsystem 101 with data signals 109received from object information retrieval subsystem 102 to deducerelevant information about objects or other subjects a subject may beinteracting during an activity or when exhibiting a certain behavior.

In a specific embodiment of this disclosure, object informationretrieval subsystem 102 continuously, periodically or otherwiseindependently from inputs from event detection subsystem 101 reads NFCtags from object that are within range of the electronic device thathouses object information retrieval subsystem 102 in its entirety or inpart. The detection of occurrence of an activity/behavior signal may beused to filter out information about specific, relevant objects that arerelated to the activity/behavior of interest. Where the objects areassociated with an event, indications of, or references to, the specificobjects might be recorded as part of a record of the event, so that theinformation can be later used for filtering or other processing.

In a specific embodiment, object information retrieval subsystem 102collects data independent from inputs it receives from the eventdetection subsystem but only sends data to processing and analysissubsystem 103 when or some time after, it receives an activation signal107 from event detection subsystem 101. Object information retrievalsubsystem 102 may send only a subset of the data it has received fromobjects over an NFC link. For example, it may only send data that itreceives in a fixed or configurable time window related to the time itreceives the activation signal 107. For example, it may only send dataimmediately preceding and/or immediately following the activation signal107. Other time windows are also possible. Object information retrievalsubsystem 102 may do additional processing on the data before sending itto processing and analysis subsystem 103.

In one embodiment of this disclosure, the object information retrievalsubsystem 102 includes a camera and the information about the object orobjects may be derived from the analysis of an image or video recordingas described in detail in patent application Vleugels II.

In a preferred embodiment of this disclosure, object informationretrieval subsystem 102 collects data from objects without anyintervention or input from the subject. In another embodiment of thisdisclosure, some user input or intervention is required. For example,the user may be prompted to move the NFC reader closer to the NFC tag ofinterest. In another example, the user may be prompted to activate theNFC reader circuitry, or portions of the NFC reader circuitry or takeone or more actions that allow the NFC reader circuitry to issue readcommand.

FIG. 2 shows a high level functional diagram of a monitoring system inaccordance with an embodiment of the present invention that requiresuser intervention. As shown in FIG. 2, one or more sensor units 300interact with event detection subsystem 301 and send sensor data toevent detection subsystem 301. Upon detection or inference of an actual,probable or imminent event, event detection subsystem sends one or morenotifications to user interaction unit 302 to request activation of NFCscan action. Notifications may be rendered to the user as a displayedtext message, as a displayed image, as an audio message, as a LEDsignal, as a vibration, etc. A combination of user interfaces may alsobe used. Other user interface may also be used. The user may respond tothe notification(s) using one or more user interfaces of the userinteraction unit 302. A user response may trigger user interaction unit302 to send a scan command 310 to object information retrieval subsystem303. Upon receiving scan command 310 or some time after receiving scancommand 310, object information retrieval subsystem 303 may activate NFCreader 306 and obtain information about object 304 over a wirelesscommunication link 311 between NFC reader 306 and NFC tag 307.Information obtained over wireless link 311 may contain informationabout brand, type, content, expiration date, lot number, etc., of object304. Other information about the object may also be retrieved. Eventdata elements 314 from event detection subsystem 301 and object dataelements 313 retrieved by object information retrieval subsystem 303from one or more objects over a wireless link 311 may be sent toprocessing and analysis subsystem 305. Additional processing may beperformed and the event and object data elements may be stored in adatabase on one or more data storage units 315.

In another example, the event detection system automatically scans forNFC tags, but upon receiving data from an object, the object informationretrieval subsystem sends a message to the subject and the subjectauthorizes, or not, transmission to a processor or analysis subsystem orthe subject confirms the information directly. This message may also besent by processing and analysis subsystem.

Processing by processing and analysis subsystem 103 may include but isnot limited to filtering, extracting specific data elements, modifyingdata or data elements, combining data or data elements obtained frommultiple objects, combing data or data elements with data obtained fromother sources not collected via NFC. Examples of filtering may includebut are not limited to filtering based on distance or estimated distancebetween object or objects and object information retrieval subsystem,based on signal strength of received NFC signals, based on order inwhich data is received from objects, based on information in the data orin specific data elements. Other filtering mechanisms or criteria usedfor filtering are also possible. Object information retrieval subsystem102 may stop sending data after a fixed time, after a configurable time,after reading a fixed or configurable number of tags. Other criteria mayalso be used. Object information retrieval subsystem 102 may send dataonly from a single object, from a subset of objects or from all objectsfor which it receives data within the specified time window.

In a different embodiment of this disclosure, object informationretrieval subsystem 102 continuously, periodically or otherwiseindependently from inputs from event detection subsystem 101 reads NFCtags from object that are within range of the electronic device thathouses object information retrieval subsystem 102 in its entirety or inpart and sends such data to processing and analysis subsystem 103independent of any signals from event detection subsystem.

Processing and analysis subsystem 103 receives data inputs from eventdetection subsystem 101 and object information retrieval subsystem 102.It is also possible that processing and analysis subsystem 103 receivesinputs only from object information retrieval subsystem 102. Processingand analysis subsystem 103 may do additional processing on the data andanalyze the data to extract information about the object(s) orsubject(s) from the data it receives. Processing may include but is notlimited to filtering, extracting specific data elements, modifying dataor data elements, combining data or data elements obtained from multipleobjects. Analysis may also include comparing specific data elements todata that is stored in a look up table or database, correlating dataelements to data elements that were obtained at an earlier time and/orfrom different subjects. Other processing and analysis steps are alsopossible. Processing and analysis subsystem 103 may store raw orprocessed data in data storage unit(s) 104. Storage may be temporary orpermanent.

In some embodiments, the outputs from processing and analysis subsystem103 may be available in real-time, either during or shortly after theevent. In other embodiments, the outputs may not be available until alater time.

Processing and analysis subsystem 103 may be implemented on the mobiledevice, on the wearable sensor device, or on another electronic device.Processing and analysis subsystem 103 may also be distributed acrossmultiple devices such as for example across the mobile device and thewearable sensor device. In another example, processing and analysissubsystem 103 may be distributed across the mobile device and a local orremote server. Processing and analysis subsystem 103 may also all beimplemented on a local or remote server. Information may be stored in asuitable format, distributed over multiple locations or centrallystored, in the form recorded, or after some level of processing. Datamay be stored temporarily or permanently. Data may be stored locally onthe wearable sensor device, on the mobile device or may be uploaded overthe Internet to a server.

The object information retrieval subsystem may be housed in its entiretyor in part inside a battery operated electronic device and it maydesirable to minimize the power consumption of the object informationretrieval subsystem. When no event is detected, the radio circuitry(e.g., the NFC reader circuitry or Bluetooth Low Energy module) may beplaced in a low power state. Upon detection or inference of an actual,probable or imminent occurrence of an event, the object informationretrieval subsystem may be placed in a higher power state. One or moreadditional circuitry inside the object information retrieval subsystemmay be powered up to activate the object information retrievalsubsystem, to improve the range or performance of object informationretrieval subsystem, etc. In one specific example, the NFC reader isdisabled or placed in a low power standby or sleep mode when no event isdetected. Upon detection or inference of an event, the NFC reader isplaced in a higher power state in which it can communicate with NFC tagsof neighboring objects. After reading a pre-configured number of NFCtags, after a pre-configured time or upon detection of the end orcompletion of the event, the NFC reader may be disabled again or may beplaced back into a low power standby or sleep mode.

Application: Food Logging

In existing food journaling methods, if a user forgets to make an entryor does not make an entry for another reason (intentionally orunintentionally), there is no record or history of the eating event.This leads to food diaries being incomplete, inaccurate and considerablyless useful for therapeutic purposes. Furthermore, content recorded inexisting food logging methods is typically limited to self-reporting ofcontent and amounts. There is no information about importantcharacteristics of how the food was consumed (e.g., pace, duration ofmeal).

The monitoring system shown in FIG. 1 may be used to automate or reducethe friction of food logging. In one embodiment of the presentdisclosure, event detection subsystem 101 uses information deduced frommotion sensors such as an accelerometer or gyroscope to detect andmonitor eating or drinking events from a subject's hand gestures. Inother embodiments, different sensor inputs may be used to deduce eatingor drinking events. Other sensors may include, but are not limited to,heart rate sensors, pressure sensors, proximity sensors, glucosesensors, optical sensors, image sensors, cameras, light meters,thermometers, ECG sensors and microphones.

Outputs of event detection subsystem 101 that indicate the occurrence ofa subject's eating or drinking events are recorded. Event detectionsubsystem 101 may do additional processing to obtain additional relevantinformation about the event such as start time, end time, metricsrepresentative of the subject's pace of eating or drinking, metricsrepresentative of quantities consumed. This additional information mayalso be recorded.

The detection of the occurrence of an eating or drinking event may berecorded as an entry in a food journal. Additional informationassociated with the eating or drinking event that can be obtained fromthe event detection subsystem may also be recorded as part of theconsumption event entry in the food journal. This can take the place ofmanually entering each eating event. The information could be stored ona remote server, integrated in a mobile app and displayed to the user,or integrated with a user's personal medical record, or shared with ahealthcare provider, insurer, caregiver or other entity or person whohas an interest or reason to receive food journaling information.Information about eating or drinking events being recorded may includetime of event, duration of event, location of event, metrics related topace of consumption, metrics related to quantities consumed, eatingmethod(s), utensils used, etc.

In another embodiment of the present disclosure, the detection of theactual, imminent or probably start of an eating or drinking event maytrigger a camera module to be activated and information about thecontent of consumed foods and drinks may be obtained from image analysisas described in Vleugels I and/or Vleugels II.

In a variation, upon detection of an eating or drinking event, an objectinformation retrieval subsystem may be activated to collect additionalinformation about the content of the foods or drinks. In one embodimentof the current disclosure, upon detection of the occurrence of an eatingor drinking event, an object information retrieval subsystemautomatically activates an NFC reader and wirelessly obtains informationabout the brand, type and/or content of foods and/or drinks beingconsumed from NFC tags that are built into or attached to the packagingof foods and/or drinks being consumed. This can take the place ofmanually entering each eating event and what was eaten. This informationis then added to the eating or drinking event records in the subject'sfood log. Other embodiments are also possible. Some user interventionmay be required to activate the NFC reader. For example, the user may beprompted to move the NFC reader closer to the NFC tag of interest. Inanother example, the user may be prompted to activate the NFC readercircuitry, or portions of the NFC reader circuitry or take one or moreactions that allow the NFC reader circuitry to issue read command.

Processing and analysis subsystem 103 may combine inputs received fromevent detection subsystem 101 and object information retrieval subsystem102 to derive information about a subject's activity or behavior relatedto an event that cannot be derived from just one of the inputs alone.For example, the data obtained from object information retrievalsubsystem 102 may provide information that the subject is eating aspecific food, but may not provide any information how much of the fooditem is consumed or how fast the specific food item is consumed. Thepace and quantity in turn may be derived from data received from eventdetection subsystem 101. By combining the information from bothsubsystem, it becomes possible to derive a more comprehensive andaccurate model of a subject's actual activities and behaviors.

In a simple embodiment, information obtained from event detectionsubsystem may be used to automatically record eating and/or drinkingevents in a food journal. Additional information about the eating ordrinking event such as start time, end time, duration, pace, andquantities consumed may also be recorded. In one example, the user maystill have to enter the content of the foods or drinks consumedmanually, but the food log uploading would provide an entry which theuser can complete. In other variations, the multiple sources could bemerged or combined in other ways.

In another example, additional processing may be performed whereby therecorded information of an eating event is compared to the informationfrom previous eating events. The comparison against historicalinformation is used to determine or predict what the subject is eating.As one example, at one point in time, the content of the food(s) beingconsumed is added to the food journal. This may be a manual entry by auser, may be based on the analysis of an image of the food, may be basedon information obtained from the read out of an NFC tag, etc. Othermethods are also possible.

In some variations, it may be beneficial to use information fromprevious consumption events to determine or predict what a user isconsuming during a new consumption event. A data storage unit may storea database with records of consumption events. Each record containsvalues of at least one or more features related to the consumption eventand at least one or more labels related to the consumption event. Aconsumption event record may include other information as well. In thisexample, the feature set comprises a “meal signature”, as itidentifies/characterizes the meal.

A dataset of raw data from sensors, wearable devices and processors forthe monitoring system might distill data down to a set of features thatare directly or indirectly related to a consumption event, thus allowingsensor data to be compressed, simplified and possibly be more easilyprocessed. Features may include day of week of the event, time of theevent, duration of the event, location of the event, metrics related topace, metrics related to quantities used or consumed, metrics related tomethods, utensils or equipment used, etc. For example if the event is aneating event, features may include metrics related to the pace ofeating, metrics related to quantities consumed, eating method(s) such aseating with spoon, fork or fingers, etc. Other features are alsopossible. Features from other non-hand gesture sensors such as heartrate sensors, motion sensors, temperature sensors, audio sensors, orblood pressure sensors are also possible. Features may also includelocation information (e.g., from a GPS system), calendar information,and inputs from applications the user uses or interacts with.

Food/drink type, brand or content information or meal descriptors mightconstitute the “labels” for the consumption event records in thedatabase. The dataset in the database may be used to train a classifierto predict food content or meal descriptors (labels) from the features.Labels for the consumption event dataset used for training may beobtained from manual entry by a user, may be based on the analysis of animage of the food, may be based on information obtained from the readout of an NFC tag, etc. Other methods are also possible.

The processing and analysis subsystem may use statistics, machinelearning or artificial intelligence techniques to determine or predictthe food(s) being consumed in the present eating event. The processingand analysis subsystem may store all previously recorded foods in adatabase and for the present eating event assign a confidence level toeach of the food(s) in the database using statistics, machine learningor artificial intelligence. Other analytics methods are also possible.Based on the confidence levels, the processing and analysis subsystemthen makes a determination as to what the food is that is being consumedor that is most likely being consumed. A confidence level might be for adecision on whether or not an event occurs, but also might be a decisionon which of multiple possibilities more likely occurred. For example,there might be that one of four events occurred and there is aconfidence score associated with each of the four events and the eventwith the highest confidence score is presumed to be the event thatoccurred.

When a new consumption event occurs or some time after the occurrence ofa new consumption event, the feature values may be derived from theevent detection subsystem. Features may be entered in a detector thatuses the trained classifier model. The detector assigns a confidencelevel to each food content label or meal descriptor in the database.

More than one food or drink may be consumed during a consumption event.Additional processing of the gestures within a single consumption eventmay be required to determine which gestures are related to what food ordrink item. For example, if a person is drinking a can of soda whileeating nuts, the consumption event may have 2 labels: “can of soda” and“nuts”. Additional processing may be required to separate gesturesrelated to drinking the can of soda from gestures related to eatingnuts. The set of features for the consumption event may include featuresthat relate specifically to the drinking soda event, features thatrelate specifically to the eating nuts event and/or features that relateto the combination of the eating and drinking event. Examples of thelatter might be time of the consumption event, total duration of theoverall consumption event, location information, etc. Other features arealso possible. In the above example, the event detection system mightdetermine that eating nuts and drinking a soda are all part of oneevent, that of having a meal or a snack.

The consumption event may be stored as a single entry in the database.The entry for the consumption event may include features that relate tothe drinking soda event, features that relate to the eating nuts eventand features that relate to the overall consumption event, includingboth the eating and drinking. The entry for the consumption event mayinclude one or more labels that specifically relates to the eating nutsevent, one or more labels that specifically relate to the drinking sodaevent and/or one or more labels that relate to the overall consumptionevent comprising eating and drinking.

In addition or instead of the entry for the overall consumption event,one or more entries may be created in the database that specificallyrelate to one of the events of “drinking soda” or “eating nuts”. Suchentries may include features that specifically relate to only one of theevents and/or features that related to the overall consumption event(e.g., time, duration, location).

In another example, a consumption event may include eating soup with aspoon and eating pasta with a fork. In this case, all gestures areeating gestures, but the gesture recognition logic may distinguish andcategorize gestures based on the eating method or utensil that was used,in this example: eating with a fork versus eating with a spoon.

The food logging system enables a high level of automation of the foodjournaling process. Once the food content associated with a consumptionevent that might comprise one or more events has been added to thedatabase, it may be used in the future to determine or predict what auser is consuming or has consumed simply based on the “meal signature”(i.e., information related to the eating activity or behavior associatedwith the event) and without requiring the user to enter manually enterthe food content again or without collecting that information through anobject information retrieval subsystem.

For example, a system may record that a user ate a bagel, along withcharacteristics of that food intake event. Characteristics may includebut are not limited to time of the event, the location of the user attime of the event, the number of bites, duration of the event, theeating method, the pace of eating, other events or behaviors occurringcoinciding or otherwise related in time to the event etc. Othercharacteristics are also possible. The system may also record that at adifferent time, the user ate yoghurt, along with characteristics of thatfood intake event. The system may record a plurality of food intakeevents, its content and one or more of its characteristics and store ashistorical information in a database. The information may be stored in adata table. This data table can be used when later an event detectionsubsystem detects a food intake event and the characteristics of thefood intake event can be compared to historical events informationstored in the data table to determine with a certain confidence levelthe content and/or amount consumed by the user during the food intakeevent.

In some variations, records in the database may be subdivided intogroups (e.g., based on meal type—breakfast, lunch, dinner, snack) and adifferent model may be used for each subgroup. Alternatively, the samemodel may be used but it may be trained using data from only one or aselective set of subgroups. In other variations, instead of a supervisedmachine learning approach (i.e., where features are specified manually),unsupervised learning may be used instead. With unsupervised learning,the classifier will autonomously generate features from the raw datasetprovided.

Information about a consumption event, obtained from the event detectionsubsystem, can be combined with information about the foods the user isconsuming, and additional processing/analysis may be performed on thecombined dataset to obtain additional information or insights about theactivity/behavior that cannot be obtained from looking at only one ofthe data sources alone.

For example, information about the type or content of foods the user isconsuming may be obtained from an object information retrievalsubsystem, but may not provide any information how much of the food itemis consumed or how fast the specific food item is being consumed. Thepace and quantity in turn may be derived from data received from eventdetection subsystem 101. By combining the information from bothsubsystems, it becomes possible to derive a more comprehensive andaccurate model of a subject's actual activities and behaviors. Forexample, it may be useful to know that a user is drinking from a can ofregular soda, but it may also be important to know how much the userdrinks. For example, it possible that the user does not finish the canof soda. Knowing how much of the can of soda the user consumes may beimportant to more accurately calculate caloric intake or sugar intake.

In the above manner, key characteristics of a consumption event, as wellas content being consumed, are recorded without or with minimal userintervention.

In specific embodiments, the systems and methods here might be used inconnection with a planned meal program or therapy that the user may havechosen, been recommended or been prescribed. In that case, a mealsignature can be used, possibly combined with other data, to monitorand/or confirm that a user is compliant with the planned meal program ortherapy. A system or program executed on a device that is local to theuser or on a remote server may analyze a user's meal signature, compareit to the meal signature of the corresponding planned meal and make adetermination as to whether or not the user has complied with theplanned meal protocol. The determination may use one or more features ofthe meal signature, such as for example, the date and time of the eatingevent, the pace of eating, the amount of food consumed, the content ofthe food consumed, they timing, amount and/or content of the fluidconsumed with the meal, the location of meal consumption, featuresrelated to the heart rate of the user before, during and/or after theeating event. Other features are also possible.

Application: Inventory Tracking/Replenishment

The monitoring system may be used to determine how much of an object hasbeen used/consumed, determine when a user may be running out of acertain object, or predict when a user will run out. This informationmay be fed into an automatic reordering or replenishment system.

The monitoring system may be used to monitor and track inventory ofobjects or items that can be linked to specific activities or behaviors.Such objects or items may include but are not limited to foods, drinks,toothpaste, tooth floss, hand soap, shampoo, lipstick, mascara,cigarettes, medications, shaving cream. Alternatively, the system mayalso be used to track inventory of living organisms such as animals orhumans.

When an event detection system detects the occurrence of one or morespecified activities or behaviors, it may notify an object informationretrieval system. The object information retrieval system can then beused to obtain information about the objects or items that the subjectis interacting with. This information may include the brand, model,content size, manufacturing date, purchase date, expiration date. Otherinformation may also be collected.

In one embodiment of the present disclosure, the processing and analysissubsystem combines information from event detection subsystem 101 withinformation from object information retrieval subsystem 102 to track howmuch a subject is consuming of a specific object and determine that asubject is running out or has run out of said object supply, or predictwhen a subject is likely to run out of said object supply.

For example, event detection subsystem detects and tracks each time auser brushes his teeth. Upon detection of a tooth brushing event, theobject information retrieval system gets activated and collectsinformation about the toothpaste being used (brand, model, tube size,etc.).

The event detection subsystem knows/tracks how often a tooth brushingoccurs and possibly also how much toothpaste is consumed during eachevent (alternatively, the system may assume a pre-configured “standardserving size” for each tooth brushing event). Information from the eventdetection subsystem and the object information retrieval system, may becombined to determine how much toothpaste has been used, is left in thetube, determine when a user is about to run out or has run out orpredict when a user is likely to run out. By monitoring how often asubject brushes his or her teeth and by collecting the brand and contentsize information of the toothpaste container, the processing andanalysis subsystem can determine that a subject is running out oftoothpaste or predict when a subject is likely to run out of toothpaste.This output of the processing and analysis subsystem can be linked to areordering or replenishment system.

Extra steps could be that the monitoring system stores a database ofobjects that a user may be using or consuming. This database may bepopulated with items retrieved from a shopping list, with items scannedin by a user (e.g., using a bar code, QR code, etc.). Other examples toimport items into a list are also possible. The content size of anobject or the number of items in a package is also recorded. Forexample, the database may be comprised of records, where each record mayinclude one or more of the following data elements: product brand,product model, content size.

Each time an event is detected, information about the object that isconsumed or used as part of the event is retrieved. A processing oranalysis engine that is part from the monitoring system verifies if theobject(s) are in the database. If an object is found in the database,the processing and analysis system computes an estimate for the amountof the object that was used or consumed during the present event. Theprocessing and analysis system may estimate the remaining content sizeby subtracting the estimated amount consumed from the amount recorded inthe database. It may then update the amount in the database to match thenew remaining content size taking into account the amount that wasconsumed or used during the most recent event.

When the amount in the database drops below a configured threshold, andalert or notification may be issued to activate a reordering orreplenishment process. For example, a notification may be sent to auser. Alternatively, the monitoring system may send a signal to acentral server that manages a user's online shopping account. Thecentral server may be 3rd party-owned server and the communicationbetween the monitoring system and the central server may be encryptedand may require one or more authentication steps. In response to areordering input from the monitoring system, a processing unitassociated with the central server may analyze the input and add therelevant object(s) to the user's shopping cart. The central server maynotify the user that an item has been added to his/her cart.Alternatively, the order may automatically be processed.

In a variation, the processing and analysis system of a monitoringsystem may use historical data about the consumption of a specificobject to predict when a user will run out of said object. For example,the processing and analysis system may look at the average rate ofchange of the content over time, or may apply higher order curve fittingto compute a function or equation that reflects the rate of change ofthe content of said object over time.

Similar examples may apply to smoking/cigarettes left in pack, drinkingfrom cans/how many cans of soda consumed, eating bars/how many barsconsumed, etc.

In another example, event detection subsystem may detect that a user issmoking. It may furthermore determine additional information such aspuff count. When a smoking event is detected, the object informationretrieval subsystem may be activated and provide information about thebrand, cigarette count, nicotine levels or other relevant informationabout the pack of cigarettes. By monitoring how often or how much asubject is smoking and by collecting the brand and cigarette countinformation of the pack of cigarettes, the processing and analysissubsystem can determine that a subject is running out of cigarettes orpredict when a subject is likely to run out of cigarettes. This outputof the processing and analysis subsystem can be linked to a reorderingor replenishment system. Other examples are also possible.

The monitoring system may be linked to an automatic or semi-automaticreordering or replenishment system. The reordering or replenishmentsystem may be at a remote location and may be implemented on a remoteserver that can be accessed via the Internet.

Application: Production Line Monitoring/QC Automation

The monitoring system described in the first section may be used forquality control purposes on a production line. Detection of a handmotion that is indicative of a production line operator performing aspecific action or executing a specific procedure. Upon detection ofsuch event, object information collection subsystem (e.g., using NFC)may collect information about objects/items that operator may beinteracting with. This could be information about parts on an assemblyline, the part number of the finished part, etc. This may be recorded ina central system that maps operator to finished part IDs, maps finishedpart to lot numbers of individual parts, etc. The central system maylink this information to data retrieved from a Quality Control system.This may provide information about performance of a specific operator.

Application: Medication Adherence

Since the event system is able to monitor events and gestures anddetermine consumption, this can be used to automatically monitor amedication administration protocol that defines when medication needs tobe taken and with what foods or other actions. This may be with specificmeal categories such breakfast, at certain times of day, etc. Amedication administration protocol might specify if patient should eator drink with medication. Optionally, the medication administrationprotocol may also specify how much food or liquid patient shouldconsume.

For example, when medication needs to be taken at certain times of day,a medication adherence system can monitor the time and when it is timeto take medication issues an alert. It may then also activate the eventdetection subsystem (if it is not yet already active) and startmonitoring the outputs of the event detection subsystem. It waits for anotification confirmation from user that he/she has taken themedication.

If a confirmation is received and if medication administration protocolprescribes that medication needs to be taken with food or liquid, themedication adherence system monitors the outputs from eating/drinkingevent detection subsystem and determines if rules specified bymedication administration protocol are met. This could be as simple asconfirming that an eating or drinking event has occurred with or shortlyafter the intake of the medication. In case the medicationadministration protocol specifies that a minimum amount of food or fluidneeds to be consumed, the medication adherence system may monitor theadditional outputs of the event detection subsystem (metrics related toquantities consumed) to confirm that this condition is met. Differentrules/logic to determine if medication administration protocol has beenmet are also possible.

If no confirmation is received, the medication adherence subsystem mayissue a second notification. Additional notifications are also possible.After a pre-configured number of notifications, the medication adherencesubsystem may issue an alert. Alert me be issued to user as a textmessage, or may be sent to a remote server over the internet or over acellular connection (e.g., to hospital, caregiver).

If medication needs to be taken with specific meal categories, themedication adherence system may monitor the outputs of an eatingdetection subsystem. When an eating event is detected, it will use logicto determine the applicable meal category. If the meal category matchesthe category described by the medication administration protocol, itwill issue a notification to remind the user to take his/her medication.If the medication administration protocol prescribes that food or liquidshould be consumed with the medication, the monitoring logic outlinedabove may be implemented to determine that the medication administrationprotocol has been adhered to.

The medication adherence system may monitor the outputs of an eatingevent detection subsystem to determine if the start of an eating eventhas occurred and determine if the eating event is of the applicable mealcategory. When an eating event is detected and the meal category matchesthe category described by the medication administration protocol,certain actions might be taken, such as that it may activate an objectinformation retrieval system (e.g., NFC tags, imaging) to collect moreinformation about the objects the user is interacting with. In this way,it may obtain information about the medication from an NFC tag attachedto the pill box or medication container. In another use, it may verifythat the medication matches the medication prescribed by the medicationadministration protocol and/or issue a notification to remind the userto take his/her medication. If the medication administration protocolprescribes that food or liquid should be consumed with the medication,the monitoring logic outlined above may be implemented to determine thatthe medication administration protocol has been adhered to.

The system might also incorporate details about the medication obtainedfrom the object information collection system or through a differentmethod in the notification, record confirmations in response to thenotification from user that he/she has taken the medication, ask theuser for additional information about the medication, and/or send theuser a follow-on question at a pre-configured time after the user hastaken the medication to get additional inputs. (e.g., query about howthe user is feeling, query about pain levels, prompt to measure bloodsugar level).

Application: Insulin Therapy/Meal-Aware Artificial Pancreas

The system might be used to detect onset of eating/drinking to startadministering micro-doses of insulin. An insulin dosage calculator cantake into account glucose level at start of eating, a slope of glucoselevel at start of eating to determine dosage and timing of insulindelivery. Insulin may be delivered at once or in multiple micro-doses.

If additional information about food is obtained from object informationretrieval subsystem (e.g., drinking a can of regular soda versus dietsoda), this information may be taken into account by the insulin dosagecalculator. For example, if a food item with high sugar content is beingconsumed, the insulin dosage calculator may increase the dosageadministered per micro-dosing event or may increase the number ofmicro-doses being delivered in a given period of time.

In a specific embodiment, the system might be used to detect onset ofeating or drinking to retrieve data from a blood glucose meter orcontinuous glucose monitoring (“CGM”) device equipped with a wirelesstechnology. The system may collect one or more glucose readings. If norecent readings are available, it may prompt or remind a user to take ameasurement. Alternatively, upon detection of the onset of eating ordrinking, the system may prompt or remind a user to take a measurement.The system may subsequently retrieve the measurement data from theglucose meter using a wireless link.

An insulin management system may use the inputs from the event detectionsystem and/or the glucose readings to make an insulin dosage decision orrecommendation. The recommendation may be communicated to the user, forexample in a notification or other message. The system may subsequentlyinteract with a connected insulin delivery device such as an insulin penor an insulin pump to confirm if the recommended dosage, a dosagedifferent from the recommended dosage or no dosage has beenadministered. The system may send another message to the user to remindthe user again or recommend a different dosage. The system may store thedata from the event detection subsystem, the glucose meter and/or theinsulin delivery devices.

Alternatively, the system may use the inputs from the event detectionsystem and/or the glucose readings and interact with a connected insulinpen or connected insulin pump to initiate an insulin dosageadministration without user intervention.

In yet another embodiment, an insulin management system may process oneor more inputs from the event detection system and one or more readingsfrom a glucose meter of CGM to determine if a patient is treating ahypoglycemic event.

Application: Control of External Devices or Objects within User'sSurroundings

As explained herein, an event detection system can send a trigger signalto an ancillary system to perform a possibly or apparently unrelatedact. As an example, when an actual, imminent or probable binge eatingevent is detected, the detection may be used to control a lightingsystem to change the tone/intensity of the light in the room topositively influence behaviors, start playing a particular genre ofmusic, a particular song, music from an artist selected by the user atan earlier time or at or near the time of the event, change thetemperature of a room, or other response that need not be specific tothe event detection system, but that has some effect that, independentof the ancillary system, is desired. As a result, it is not required forthe ancillary system to be aware of why various actions are beingtriggered and the ancillary system does not need to be specificallyprogrammed or designed to work as part of an event response and feedbacksystem.

These interactions with connected external objects can be seen asbehavioral interventions, but can happen autonomously orsemi-autonomously, to create more advanced experiential interventionsthat are personalized and contextualized.

Application: Dynamic Adjustment of Interventions

The detection of the start of an event (e.g., eating, drinking, smokingevent) or of certain characteristics or behaviors associated with theevent (e.g., pace of eating, pace of inhales when smoking) can trigger amessage to be sent to the user. The message can, for example, be sent asa notification on the user's wearable device. The user may be notifiedthat a message is being sent by means of an audio signal or by means ofhaptic feedback. Other mechanisms to inform the user that a message hasbeen sent are also possible.

The message can have an educational or promotional content, be a prompt,a reminder, a coaching tip, etc. In some embodiments, the user canconfirm that he or she has received and/or read the message, for examplewith a button press, by means of selection of a displayed action buttonon a touch display on the wearable device, by means of a voice inputusing a microphone, or with a specific hand movement that is understoodby the system as a user confirmation.

In other embodiments, the system may infer that the user is reading orhas read the message without explicit user confirmation. For example, ifthe message is displayed on a user's wristworn wearable device, a systemmay analyze the user's wrist movement and/or rotation following thesending and display of the message, e.g., by analyzing the data from oneor more motion sensors in the wearable device, to determine if and/orfor how long the user is looking at his/her display on the wearablefollowing the sending of the message. From this analysis, the system canthen determine or infer if the user is reading or likely reading, or hasread or is likely to have read the message.

The message can also be a request for user input. As an example, when astart of an eating event is detected, the system may send a message toask the user to rate his hunger level, assess the healthiness of hisfood choices, or assess her emotional state. The system may recordwhether or not the user responds, the actual user response, and the timeit took for the user to respond.

Explicit user confirmation or user responses as well as inferred useractions (such as inferring that the user read or likely read themessage) can be recorded and analyzed to measure the level of userengagement or changes in level of user engagement at a given time, orover a certain time window. The data can be analyzed to determine thelevel of user engagement to messages in general, and/or to assess thelevel of user engagement in response to one or more specific messagetypes, specific message content, specific message intensity or specificmessage tone. The information and insights about a user's engagementwith messages in general or with specific messages or message types canbe recorded in a database that records data and information about auser, and can be used to adjust the content, tone, intensity, etc. offuture messages and user interventions.

Survey Component

The event detection system might include a survey component. Since theevent detection system knows when an event is going to occur or isoccurring, the event detection system can time a survey question for theuser. For example, the event detection system might ask the user justbefore an eating or smoking event starts or shortly after it starts toanswer a survey question, such as “what were your emotions before youstarted this event?” or “what triggered you to start this event?” andthus provide just-in-time impressions of the user that the user,caregiver, or software program or healthcare maintenance or diseasemanagement system can later review or process to learn more aboutassociations between events and states of mind. This is likely to bemore accurate than, say, a weekly e-mail survey asking the user “wheneach of the events in the last week began, what were you feeling?”.Since the event detection system is accurately predicting and detectingparticular events, and recording their timestamps, it can easily presentreal-time, in-the-moment surveys.

User Engagement

A feedback system might assess the level of user engagement to evaluatewhat message types, content, intensity, or tone a user best responds to.This information can then be used by the feedback system to furthercustomize the messages for the specific user.

In other variants, the assessment of level of user engagement at a giventime or over a time window, optionally combined with additionalinformation such as for example the user's responses to certainmessages, or data retrieved from other applications such as a user'scalendar, can used by a feedback system to assess a user's motivationand/or a user's readiness for acting on the information delivered in themessage. A feedback system may use this to dynamically adjust the type,content, tone or intensity/frequency of its messages. For example, if afeedback system determines that a user is not motivated or ready to act(e.g., the user might be distracted by another event developing in hislife), the feedback system may decide to scale back on the frequency ofthe messages it sends, to soften the tone of the messages, to change thecontent of one or more messages. On the other hand, when a feedbacksystem determines that a user is highly engaged, as inferred from a highlevel of engagement with messages sent to the user, it may increase thefrequency of the messages, increase the complexity level of one or morecoaching tips, etc.

FIG. 3 shows a high level functional diagram of a dietary tracking andfeedback system in accordance with an embodiment of the presentinvention. A system for dietary tracking and feedback may in partinclude one or more of the following: a food intake event detectionsubsystem 501, one or more sensors 502, a tracking and processingsubsystem 503, a feedback subsystem 506, one or more data storage units504 and a learning subsystem that might perform non-real-time analysis.In some embodiments, elements shown in FIG. 3 are implemented inelectronic hardware, while in others some elements are implemented insoftware and executed by a processor. Some functions might sharehardware and processor/memory resources and some functions might bedistributed. Functionality might be fully implemented in a sensordevice, or functionality might be implemented across the sensor device,a processing system that the sensor device communicates with, such as asmartphone, and/or a server system that handles some functionalityremote from the sensor device. For example, a wearable sensor devicemight make measurements and communicate them to a mobile device, whichthen uploads them over the Internet to a server that further processesthe data. Data or other information may be stored in a suitable format,distributed over multiple locations or centrally stored, in the formrecorded, or after some level of processing. Data may be storedtemporarily or permanently.

A first component of the system illustrated in FIG. 3 is the food intakeevent detection subsystem 501. The role of this subsystem is to identifythe start and/or end of a food intake event and communicate an actual,probable or imminent occurrence of the start and/or end of a food intakeevent to other components in the system.

In general, the device detects what could be the start of a food intakeevent or the probable start of a food intake event, but the device wouldwork sufficient for its purposes so long as the device reasonablydetermines such start/probable start. For clarity, that detection isreferred to as a “deemed start” of a food intake event and when variousprocesses, operations and elements are to perform some action orbehavior in connection with the start of a food intake event, it wouldbe acceptable for those various processes, operations and elements totake a deemed start as the start even if occasionally the deemed startis not in fact a start of a food intake event.

In one embodiment, the detection and/or signaling of the occurrence ofthe deemed start of a food intake event coincides with the deemed startof a food intake event. In another embodiment, it may occur sometimeafter the deemed start of the food intake event. In yet anotherembodiment, it may occur sometime before the deemed start of the foodintake event. It is usually desirable that the signaling is close to thedeemed start of the food intake event. In some embodiments of thecurrent disclosure, it may be beneficial that the detection and/orsignaling of the deemed start of a food intake event occurs ahead of thestart of said food intake event. This may for example be useful if amessage or signal is to be sent to the user, a healthcare provider orcaregiver ahead of the start of the food intake event as a coachingmechanism to help steer a user's food intake decisions or eating habits.

In a preferred embodiment of the present disclosure, the detection ofthe start and/or ending of a food intake event by the food intake eventdetection subsystem 501 happens autonomously and does not require anyspecial user intervention. To accomplish this, the food intake eventdetection subsystem may use inputs 507 from one or more sensors 502.Sensors may include, but are not limited to, accelerometers, gyroscopes,magnetometers, magnetic angular rate and gravity (MARG) sensors, heartrate sensors, blood pressure sensors, image sensors, cameras, opticalsensors, proximity sensors, pressure sensors, odor sensors, gas sensors,glucose sensors, Global Positioning Systems (GPS), and microphones.

Methods for autonomous detection may include, but are not limited to,detection based on monitoring of movement or position of the body or ofspecific parts of the body, monitoring of arm movement, position orgestures, monitoring of hand movement, position or gestures, monitoringof heart rate, monitoring of blood pressure, monitoring of fingermovement, position or gestures, monitoring of swallowing patterns,monitoring of mouth and lips movement, monitoring of saliva, monitoringof movement of cheeks or jaws, monitoring of biting or teeth grinding,monitoring of signals from the mouth, the throat and the digestivesystem. Methods for detection may include visual, audio or any othertypes of sensory monitoring of the person and/or his or hersurroundings. The monitored signals may be generated by the dietarytracking and feedback system. Alternatively, they may be generated by aseparate system but be accessible to the dietary tracking and feedbacksystem through an interface. Machine learning and other data analyticstechniques may be applied to detect the start or probable start of afood intake event from the input signals being monitored.

In one example, the food intake detection system 501 may monitor theoutputs of accelerometer and/or gyroscope sensors to detect a possiblebite gesture or a possible sip gesture. Such gestures might bedetermined by a gesture processor that uses machine learning to distillgestures from sensor readings. The gesture processor might be part ofthe processor of the worn device or in another part of the system.

Gesture detection machine learning techniques as described elsewhereherein may be used to detect a bite gesture or sip gesture, but othertechniques are also possible. The food intake detection system 501 mayfurther assign a confidence level to the detected bite gesture or sipgesture. The confidence level corresponds to the likelihood that thedetected gesture is indeed a bite or sip gesture. The food intakedetection system may determine that the start of a food intake event hasoccurred based on the detection of a gesture and its confidence levelwithout any additional inputs. For example, the food intake eventdetection system 501 may decide that the start of a food intake eventhas occurred when the confidence level of the bite or sip gestureexceeds a pre-configured threshold.

Alternatively, when a possible bite or sip gesture has been detected,the food intake event detection system 501 may use additional inputs todetermine that the start or probable start of a food intake event hasoccurred. In one example, the food intake event detection system 501 maymonitor other gestures that are close in time to determine if the startof a food intake event has occurred. For example, upon detection of apossible bite gesture, the food intake event detection system 501 maywait for the detection of another bite gesture within a certain timewindow following the detection of the first gesture and/or with acertain confidence level before determining that the start of a foodintake event had occurred.

Upon such detection, the food intake detection system 501 may place oneor more circuits or components into a higher performance mode to furtherimprove the accuracy of the gesture detection. In another example, thefood intake event detection system 501 may take into consideration thetime of the day, or the location of the user to determine if the startor probable start of a food intake event has taken place. The foodintake event detection system may use machine learning or other dataanalytics techniques to improve the accuracy and reliability of itsdetection capabilities. For example, training data obtained from theuser and/or from other users at an earlier time may be used to train aclassifier. Training data may be obtained by asking for userconfirmation when a possible bite or sip gesture has been detected. Alabeled data record can then be created and stored in memory readable bythe gesture processor that includes the features related to the gesture,along with other contextual features, such as time of day or location. Aclassifier can then be trained on a labeled dataset comprised ofmultiple labeled data records set of labeled data records, and thetrained classifier model can then be used in a food intake eventdetection system to more accurately detect the start of a food intakeevent.

In another embodiment, the food intake detection subsystem may usetriggers to autonomously predict the probable start of a food intakeevent. Methods for autonomous detection of a probable start of a foodintake event based on triggers may include, but are not limited to,monitoring of a person's sleep patterns, monitoring of a person's stresslevel, monitoring of a person's activity level, monitoring of a person'slocation, monitoring of the people surrounding a person, monitoring of aperson's vital signs, monitoring of a person's hydration level,monitoring of a person's fatigue level, monitoring of a person's heartrate. In some cases, the food intake detection subsystem may monitor oneor more specific trigger signals or trigger events over a longer periodof time and, in combination with the non-real-time analysis and learningsubsystem 505 apply machine learning or other data analytics techniquesto predict the probable occurrence of a start of a food intake event.

For example, without any additional information, it can be verydifficult to predict when a user will eat breakfast. However, if thesystem has a record over a number of days of one or more of thefollowing: the user's wake up time and the day of the week, of a user'seating times and the day of the week, of a user's physical activity,heart rate, body temperature and/or blood pressure preceding eatingtimes and the day of the week, the system can use that historicalpattern in determining a likely time for the user to eat breakfast.Those records might be determined by the system, possibly with feedbackfrom the user about their accuracy or those records might be determinedby the user and input via a user interface of the system. The userinterface might be the worn device itself or, for example, a smartphoneapp. As a result, the system can process correlations in the historicaldata to predict the time or time window that the user is most likely tohave breakfast based on one or more of the following: the current day ofweek, the current time, at what time the user woke up, the user'scurrent or recent heart rate, the user's current or recent bloodpressure, the user's current or recent physical activity, the user'scurrent or recent body temperature. Other trigger signals or triggerevents may also be used by the non-real-time analysis and learningsubsystem 505 to predict the time that a user will eat breakfast.

In another example, the non-real-time analysis and learning system 505may, over a certain period of time record the stress level of a user.The stress level may, for example, be determined by monitoring andanalyzing the user's heart rate or certain parameters related to theuser's heart rate. The stress level may also be determined by analyzinga user's voice. The stress level may also be determined by analyzing thecontent of a user's messages or electronic communication. Other methodsfor determining the stress level are also possible. The non-real-timeanalysis and learning system 505 may furthermore, over the same periodof time, record the occurrence of food intake events and certaincharacteristics of the food intake event such as the pace of eating, thequantity of food consumed, the time spacing between food intake events,etc. It may then be possible by analyzing the historical data of stresslevels, the occurrence of food intake events and food intake eventcharacteristics and by looking at correlations in the historical data ofstress levels, the occurrence of food intake events and food intakeevent characteristics, to predict based on the current stress level theprobability that a user will start a food intake event in a certain timewindow in the future, or predict what time window in the future, theuser will be most likely to start a food intake event. It may also bepossible to predict characteristics of said food intake event, such asfor example pace of eating or quantity of consumption.

Predictive Behavior Signaling System

FIG. 4 illustrates an example of a behavior event prediction system thattrains a predictor subsystem from past behavior events and past behaviorindicator inputs to form a model and uses that model to predict orestimate a future behavior event from current behavior indicators. Thebehavior indicators have associated timestamps, as do the past behaviorevents, and the behavior event prediction system can provide anestimated time for occurrence of a future behavior event. Behaviorindicators of a user can be derived from sensor inputs of sensors wornby the user, environmental sensors of the user's environment,interactions of the user with technological devices, and informationprocessed, or likely processed by the user. By tracking the relativetimes of the past behavior indicators and the past behavior events,future behavior events could be predicted from current behaviorindicators.

As shown there, data inputs from external sources, data inputs 451, arereceived by a data processing subsystem 454 and are tagged with theirtime of measurement or recording by the external data source. Forexample, if the data input are sensor readings from an external sensor,then those sensor readings are tagged with the time of the actualmeasurement by the sensor, a time close to the actual measurement of thesensor or a time related to the actual measurement of the sensor. Theraw data from one or more sensor units may be sent directly to thepredictive behavior signaling system. Alternatively, a processing unitexternal to the predictive behavior signaling system may first processthe data before sending it to predictive behavior signaling system.

As shown in FIG. 4, a data processing subsystem 454 takes in behaviorevent inputs 450 and behavior indicator inputs 451 and stores it as ahistorical data set 452 that might be organized to include a data tableas shown in FIG. 4 with the behavior indicator inputs tagged with a timerelative to a corresponding behavior event and possibly a correspondingbehavior type, if the system is handling multiple event types.

Note that the data table retains relative time data for indicators andevents. For example, in the first data record, the behavior indicatorsare known to have occurred around five minutes prior to a behaviorevent. This data can be used to train a classifier model or otherwisedefine rules that the predictor subsystem will use to make a prediction(and possibly assign a confidence level to the prediction). This datawould be passed from data processing subsystem 454 to training or rulesgeneration subsystem 453 that might output a model or model parametervalues 461 for a predictor subsystem 455.

Predictor subsystem 455 might take as input current behavior indicatorinputs from which features for the data records in the data table werederived and apply the model or rules to those inputs to create an outputprediction, possibly with an assigned confidence level, of a predictedfuture behavior event. Other characteristics of the prediction may alsobe included, e.g. if the prediction is that the user will start to eatin 20 minutes, the prediction output may also include how much the useris likely to eat or for how long he is likely to eat.

The output of a behavior event detection or prediction system may feedinto a feedback system. The feedback system may have a set of rules andmessages. Based on the output of the predictor subsystem, it may selectone or more messages to feedback to a user or signal to another system.

An illustrative example of a predictive behavioral signaling system isshown in FIG. 4, as might be used to predict behavior as part ofdetecting events in advance, where those events relate to userbehaviors. The system records occurrences of a behavior event ofinterest upon receiving behavior event inputs 450, and those inputsinclude timing information of the behavior event of interest. The systemmay also record additional characteristics of the behavior events.

For example, if the behavior event of interest is food intake, thesystem may record timing information of food take event occurrences(e.g., start time, duration, end time). The system may also recordadditional characteristics of the occurrence (e.g., characteristicsinformative or indicative of pace of eating or drinking, number of bitesor sips or other characteristics informative or indicative of amountsconsumed, characteristics informative or indicative of content of mealor drink). In another example, the behavior of interest is smoking andthe system may record timing information of smoking events (e.g., starttime, duration, end time). The system may also record additionalcharacteristics of the occurrence (e.g., characteristics informative orindicative of pace of smoking, number of inhales or othercharacteristics informative or indicative of amounts inhaled,characteristics informative or indicative of nicotine content). Otherexamples of behavior events are described elsewhere herein. Otherexamples are also possible.

The system can record data inputs 451 from external data sources andoccurrences of behavior events of interest over a certain period oftime. The duration of recording can vary, but it should be sufficientlylong to create a training set. For example, it may record it over theduration of a week, a month, a year. It is also possible that itmonitors and records on an on-going basis.

The data processing subsystem 454 may process one or more inputs fromone or more external data sources and one or more records of behaviorevent occurrences to create a labeled data record of a training data setstored in data table 452. The data processing subsystem 454 aligns intime the inputs from external data sources and the occurrences of thebehavior event of interest. The data processing subsystem 454 defines atime or a time window over which it will process data inputs fromexternal sources. In one embodiment, the time or time window isreferenced relative to the occurrence of a behavior event of interest.For example, the data processing subsystem may process data inputs fromexternal data sources in a fixed time window leading up to the start ofa behavior event occurrence (e.g., a 15-minute window 1 hour before thestart of a behavior event occurrence). A plurality of time windows isalso possible. (e.g., a 5-minute window, a 10-minute window, a 15-minutewindow and a 30-minute window with those time windows all having astarting, centered or ending time 1 hour before the start of a behaviorevent occurrence). Other examples are also possible. In anotherembodiment, the data processing subsystem may define one or morespecific time or time window relative to the end of a behavior eventoccurrence. Other time references of the behavior event of interest arealso possible. In another embodiment, the time or time windows are notspecified relative to the occurrence of a behavior event of interest.Instead, they may be specified against an absolute time reference.

The data processing subsystem 454 processes data inputs from externaldata sources at the specified times or within the specified time windowsto compute one or a plurality of characteristics or “feature values” forthe labeled data record. For example, the external data source may be aset of timestamped heart rate sensor readings and the data processingsubsystem 454 may use as features the minimum, maximum, average and/orchanges in heart rate at the specified time or in the specified timewindows relative to the behavior event occurrence. In another example,the external dataset may be a log of calendar entries and the dataprocessing subsystem 454 may filter calendar entries that fall withinone or more specified time windows relative to the occurrence of abehavior event of interest. The data processing subsystem 454 may doadditional processing of those entries. For example, the data processingsubsystem 454 may filter entries, or select specific information fromthose entries.

The data processing subsystem 454 may use the processed data as featurevalues in the labeled data record of the training data set.

The data processing subsystem 454 further assigns one or more labels tothe labeled data record. Labels may have been assigned manually by ahuman or may have been generated automatically or semi-automatically. Ina specific embodiment where a time or time window relative to theoccurrence of a behavior event is used, a label can for example be thevalue of the time reference used to generate the features. For example,if the features are generated in a 15-minute time window 1 hour relativeto the start of the behavior event of interest, a label value of “60minutes” or “1 hour” may be assigned to the label of that data record.In a different embodiment, for example when the time or time windows forfeature generation are defined on an absolute time reference, a labelmay be a binary value indicative of whether or not the start of thebehavior event of interest occurred at a specific time or within aspecific time window following the time or time window over which thefeatures were generated. Additional characteristics of the behaviorevent occurrence may also be used as labels. For example, if thebehavior event is a food intake event, the pace of eating or the amountconsumed may be added as labels. In yet another example, othercharacteristics such as whether or not the food intake event was a bingeeating event may be added as a label. Other characteristics are alsopossible.

A labeled data record can be created and stored in memory readable by aprocessor.

The data processing subsystem 454 creates a plurality of labeled datarecords. For example, it may use a sliding time window(s) to generate aset of labeled data records. A classifier can then be trained on alabeled dataset comprised of multiple labeled data records, and thetrained classifier model can then be used in a predictive behaviorsignaling system to predict the start of a behavior event of interest.Multiple labeled data records may be stored in a data table.

The system further includes a predictor subsystem 455. The predictorsubsystem 455 processes later data inputs 456 from external data sourcesto generate the feature values for data inputs 456. The predictorsubsystem 455 analyzes the characteristics or “feature values” of newdata inputs 456 to generate a prediction output. For example, thepredictor subsystem 455 may feed the feature values into a trainedclassifier model to generate a prediction output. In another example,predictor subsystem 455 may apply a rules analyzer with rules derived bya rules generation subsystem 453. Other examples are also possible. Inone specific embodiment, the output signal 457 might be a prediction oftime until the start of the next occurrence of a behavior event ofinterest. In another embodiment, the prediction output signal 457 may bea binary output whether or not a behavior event of interest will happenat or within a specified time. In yet another embodiment, the predictionoutput signal 457 may be a confidence level or probability that abehavior event of interest will happen at or within a given time. In yetanother embodiment of the disclosure, the prediction output isprediction of time until the start of the next occurrence of a behaviorevent of interest along with a confidence level of the prediction. Inyet another embodiment, the prediction output is prediction of timeuntil the start of the next occurrence of a behavior event of interestalong with additional characteristics of the predicted event. If thebehavior event is a food intake event, the additional characteristicsmay for example be the predicted duration of food intake, the predictedamount of food consumption, etc. Other examples are also possible. Thepredictor subsystem 455 may process data inputs from external sources inreal-time or near real-time. Alternatively, there may be a delay betweenthe time the data inputs are generated, fed into the system and/orprocessed by the prediction subsystem.

A behavior event of interest can be a food intake event, a smokingevent, a dental hygiene event, a medication intake event, a personalhygiene event, a physical activity event, a stress indicating event suchas for example nail biting, making a phone call, driving a car. Otherexamples are also possible.

In one specific embodiment, the system predicts the occurrence of abehavior event. In other embodiments, the system may predict theoccurrence of a behavior event with specific characteristics. The systemmay for example be used to predict a binge eating episode or stresssnacking. In another example, the system may be used to predictunderhydration or malnutrition.

The above description assumes a training subsystem that uses datarecords with features and labels to train a classifier model.Alternatively, a neural network may be used to autonomously derivecharacteristics and parameters of the data inputs without explicitfeature generation. It is also possible that a different system is usedto generate one or more rules to be used by the predictor subsystem todetermine one or more predictor output signals.

Output signals 457 may be used by feedback subsystem 458 to trigger amessage to a user or a signal to an ancillary system. For example, whena predictor subsystem predicts that a user will likely incur a bingeeating episode in the next 15 minutes, the output 457 may be used byfeedback subsystem 458 to send a message or instruction 459 to a roomenvironmental control system to instruct the room environmental controlsystem to cool the room by three degrees. In the more general case,feedback subsystem 458 outputs output messages 459 to users and/oroutputs output signals 460 to ancillary devices, each in response to atrigger based on a behavior event or a predicted behavior event. Inanother example, when a predictor subsystem predicts that a user willlikely lit up a cigarette and start smoking in the next half hour, thismay be signaled to a feedback subsystem 458 through predictor outputsignal 457, and feedback subsystem 458 may send a message to the user,for example to invite the user to reach out to a caregiver to get help.

In a specific embodiment of the disclosure, a non-real time analysis andlearning system is part of a predictive meal signaling system. Thenon-real time analysis and learning system may record data from one ormore data sources along with the occurrence of and characteristics of auser's food intake events over a period of time. The data sources mayfor example be heart rate, hydration level, stress level. But other datasources are also possible. The period of time over which data iscollected may vary as well. It could for example be over several days,several weeks, several months. The collection of data may also beon-going, and not be limited to a specific period of time. In that case,the new data may be fed into the system periodically or from time totime to improve, or otherwise modify or adjust certain parameters or theaccuracy of the system. As an example, the non-real time analysis andlearning system may record a user's data such as heart rate, stresslevel, hydration level, body temperature, transpiration level, etc. overtime. It may process and analyze that data or one or more subsets ofthat data over one or more specific time windows referenced in time tothe start of a food intake event, relative to the end of a food intakeevent, and/or relative to other timing characteristics of the foodintake event. For example, a non-real time analysis and learning systemmay analyze the heart rate data in a time window preceding the start ofa food intake event.

In an example described below, there is a set of sensors, or at leastinputs from a set of sensors, coupled to a machine classification systemthat outputs gesture data from sensor readings, taking into accountrules and stored data derived from training the machine classificationsystem. A training subsystem might be used to train the machineclassification system and thereby forming the stored data derived fromtraining. Each of these components might use distinct hardware, orshared hardware, and can be localized and/or remote. In general, when agesture is detected, a system can analyze that gesture, determine likelyactual, probable or imminent activities or behaviors and provide theuser feedback with respect to those activities or behaviors. Forexample, a vibration as a feedback signal to indicate that the user haspreviously set up the system to alert the user when the user has beendrinking for a semi-continuous period of more than 45 minutes or thatthe user has reached their target for the amount of walking to be donein one session.

Various rule sets could be employed that map behavior events tosignals/messages output by a system that predicts or detects behaviorevents so that the system might output control signals to ancillarydevices that will perform actions that prevent certain behaviors thatwere predicted, or that are desired responses to certain behaviors, orthat send messages to the user or others to prepare for possiblepredicted behavior.

The behavior prediction system might determine when to send a signal ora message that relates to detected or predicted behavior events. In someembodiments, historical data is used, whereas in other embodiments, thehistorical data is analyzed in advance and distilled down to rules thatcan be stored in a messages and rules database. As one example, if theevent detection or prediction system were to detect or predict the startof a smoking event and there was historical data that includedreferences to receipt of a telephone call from a particular number atsome time before a smoking event, that might be distilled down to “Sendthe user a message two minutes after ending a call with person P1 saying‘You wanted to be notified of triggers in your efforts to reducesmoking. You typically start smoking four minutes after receiving a callfrom P1. That was two minutes ago.’ and record the user's response.”

In either case, whether the processing is done in advance or at the timeof sending a signal/message, the data table might contain parametersthat correspond behavior indicators from sensed activity orenvironmental details, a timestamp of when those parameters occurred andan indication (a timestamp or a time relative to the timestamp of thesensed activity or details) of when the event being triggered orinitiated might occur. Since the data table includes not only the itembut an indication of the timing of the item relative to the event thatthey precede, this can be used to develop, with varying levels ofconfidence, the prediction that a particular event is about to start.For example, if a heart rate sensor detects a rapid increase in heartrate for a short duration without significant body motion of the userand also detects an eating event starting 20 minutes thereafter and thedata table 406 contains several data items corresponding to behaviorindicators, such as “short heart rate burst; 20 minutes prior”, “shortheart rate burst; 21 minutes prior”, and “short heart rate burst; 20minutes prior”, from previous eating events that followed heart ratebursts, when a future heart rate burst occurs, the event detectionsystem might flag that behavior indicator as an advance signal of aneating event that may start 20 minutes in the future from that behaviorindicator's timestamp. The actual value presented for the predictedstart of the eating event need not be exact, but could be useful even ifthe actual eating event occurred at a slightly different time.

The prediction could be used to trigger a message to the user or asignal to an ancillary system. For example, where the heart rate burstis known to be related to stress, the event detection system might sendout a signal to a room environmental control system 10 minutes after thebehavior indicator (an estimated 10 minutes before an expected eatingevent start) to instruct the room environmental control system to coolthe room by three degrees.

In this manner, the data table could grow to a large dataset and mightbe combined with datasets of other users to identify patterns for use inpredicting possible events. The behavior indicators might relate togestures detected (user's hand unscrewing a bottle top, picking up autensil, etc.), biometric readings (breathing, heart rate, skintemperature), location, computer interactions (read an e-mail from aparticular person, was on a particular social media website, etc.),lighting, time of day. Each of the behavior indicators has a timestampand, following analysis and processing, an estimated time for a futureevent, such as an eating event. From several such behavior indicators,the event detection system might be able to determine when the eventwill likely start and possibly a confidence value as to how certain theevent is to occur around that time. As explained elsewhere herein, thatinformation can result in a message or signal to an ancillary device torespond to the event.

In some embodiments of the present disclosure, the machineclassification system may include additional subsystems or modifiedversions of the subsystems shown in FIG. 4. For example, if the behaviorof interest is food intake, then the data input may include data recordsof a user's food intake events, with characteristics of each event.Characteristics may be a timestamp corresponding to the start of thefood intake event, information related to the duration of the foodintake event, average pace of eating, etc. Other characteristics arealso possible.

The behavior indicators may include one or more timestamped data inputsfrom external data sources that might relate to eating events or othertypes of events that are events to be detected. Such data might becomputerized information, such calendar details like meetings andactivities extracted from one or more calendars of the user, such as thetime, duration, location, subject or agenda, classification (personal,business, etc.), participant details, telephone details (user's phonecalls such as the time, duration, phone number(s), name(s) or otherinformation of caller or call participant(s)). Other info might beinformation derived from or inferred from phone calls or other recordedconversations such as insights derived or inferred from the tone,intensity, choice of words, emotions, volume, etc. Yet other informationmight include location information, e.g., information obtained from aGPS system, inputs from one or more sensor units, including but notlimited to body temperature sensor, heart rate sensors, blood glucosesensors, sensors measuring transpiration, information about a user'sphysical activity, mental health or stress level, or sleep, informationabout a user's activity or behaviors, including but not limited to foodintake, smoking, dental hygiene activities such as flossing or brushingteeth, personal hygiene, driving a vehicle, making a phone call. Otherdata sources are also possible.

Data Structures for Data Analysis Used for Predicting Events

The notifications and interactions with the user can be customizedaccording to the timing of events as the event detection system is ableto, with limited inputs, determine the start of an event, or the futuretime of an event starting, with some confidence level, and take actionaccordingly. For example, if the event detection system determines witha certain confidence level that at a time 15 minutes in the future, theuser might begin a binge eating event, the event detection system mightoutput a message or a signal to an ancillary system in order to alterthe occurrence of the predicted event(s). In another example, if theevent detection system determines with a certain confidence level thatan eating event has begun, it can signal an ancillary system such as aninsulin microdosing apparatus to begin releasing insulin. This would bean improvement over a messaging system that uses fixed times forpresenting user messages or that only sends signals much later than thestart of an event.

In the more general case, for behavior events, there is an anchor time,which might be the start time, the end time, a peak time or another timerelative to the event, from which responses to the event are timed. Forexample, a response for a behavior event might be emitted zero minutesor three minutes after the anchor time of the behavior event and theanchor time might be the start of the event or two minutes after thestart of the event. Instead of minutes being the unit of measure of theresponse time or the anchor time, other units might be used, such asnumber of bites, number of sips, number of inhales, etc.

Additional Details

In specific embodiments, the non-real time analysis and learningsubsystem may use historical data from different users, or a combinationof data from other users and from the wearer, and use similaritiesbetween one or more of the different users and the wearer, such as age,gender, medical conditions, etc., to predict the probable start of afood intake event by the wearer.

In yet other examples, the non-real-time analysis and learning subsystem505 may use methods similar to the methods described herein to predictwhen a user is most likely to relapse in a binge eating episode or ismost likely to start stress snacking or convenience snacking.

Sensors and Machine Learning

A variety of sensors may be used for such monitoring. The monitoredsignals may be generated by the dietary tracking and feedback system.Alternatively, they may be generated by a separate system but beaccessible to the dietary tracking and feedback system for processingand/or use as trigger signals. Machine learning and other data analyticstechniques may also be applied to predict some other characteristics ofthe probable intake event, such as the type and/or amount of food thatwill likely be consumed, the pace at which a person will likely beeating, the level of satisfaction a person will have from consuming thefood, etc.

The machine learning process performed as part of gesture recognitionmight use external data to further refine its decisions. This might bedone by non-real-time analysis and learning subsystem process. The dataanalytics process might, for example, consider the food intake eventsdetected by the gesture-sensing based food intake detection system andthe gesture-sensing based tracking and processing system, thus forming asecond layer of machine learning. For example, over a period of time,food intake events and characteristics related to those food intakeevents are recorded, such as eating pace, quantity of food consumption,food content, etc., while also tracking other parameters that are notdirectly, or perhaps not obviously, linked to the food intake event.This could be, for example, location information, time of day a personwakes up, stress level, heart rate, blood pressure, certain patterns ina person's sleeping behavior, calendar event details including time,event location and participant lists, phone call information includingtime, duration, phone number, etc., email meta-data such as time,duration, sender, etc. The data analytics process then identifiespatterns and correlations. For example, it may determine a correlationbetween the number of calendar events during the day and thecharacteristics of the food intake event(s) in the evening. This mightbe due to the user being more likely to start snacking when arrivinghome, or that dinner is larger and/or more rushed when the number ofcalendar event(s) for that day exceeds a certain threshold. Withsubsystem 505, it becomes possible to predict food intake events andcharacteristics from other signals and events that are not obviouslylinked to food intake. Additional contextual metadata such as location,calendar information, day of week or time of day may be used by theprocessing and analysis subsystem to make such a determination orprediction.

Processing and analysis of one or more sensor inputs, and/or one or moreimages over longer periods of time, optionally using machine learning orother data analytics techniques may also be used to estimate theduration of a food intake event or may be used to predict that the endof a food intake event is probable or imminent.

In another embodiment, some user input 508 may be necessary or desirableto properly or more accurately detect the start and/or end of a foodintake event. Such user input may be provided in addition to externalinputs and inputs received from sensors 502. Alternatively, one or moreuser inputs may be used instead of any sensor inputs. User inputs mayinclude, but are not limited to activating a device, pressing a button,touching or moving a device or a specific portion of a device, taking apicture, issuing a voice command, making a selection on a screen orentering information using hardware and/or software that may include butis not limited to a keyboard, a touchscreen or voice-recognitiontechnology. If one or more user inputs are required, it is importantthat the user interaction is conceived and implemented in a way thatminimizes the negative impact on a person's normal activities or socialinteractions.

A food intake event detection subsystem may combine multiple methods toautonomously detect predict the actual, probably or imminent startand/or end of a food intake event.

Another component of the system is the tracking and processing subsystem503. In a preferred embodiment of the present disclosure, this subsysteminterfaces 509 with the food intake event detection subsystem 501, andgets activated when it receives a signal from the food intake eventdetection subsystem that the actual, probable or imminent start of anevent has been detected, and gets disabled when or sometime after itreceives a signal from the food intake event detection subsystem thatthe actual, probable or imminent ending of an event has been detected.Upon detection of the start of a food intake event, the device mighttrigger activation of other sensors or components of the food intaketracking system, and might also trigger the deactivation of those upondetection of the end of the food intake event.

In another embodiment of the current disclosure, the tracking andprocessing subsystem may be activated and/or deactivated independent ofany signals from the food intake detection subsystem. It is alsopossible that certain parameters be tracked and/or processedindependently of any signals from the food intake detection subsystem,whereas the tracking and/or processing of other parameters may only beinitiated upon receiving a signal from the food intake event detectionsubsystem.

FIG. 5 illustrates some of the components disposed in an electronicdevice, in accordance with one embodiment. The electronic devicetypically includes, in part, one or more sensor units 527, a processingunit 528, memory 529, a clock or crystal 530, radio circuitry 534, and apower management unit (“PMU”) 531. The electronic device may alsoinclude one or more camera modules 526, one or more stimulus units 533and one or more user interfaces 532. Although not shown, othercomponents like capacitors, resistors, inductors may also be included insaid the electronic device. Power Management unit 531 may, among otherthings, include one or more of the following: battery, chargingcircuitry, regulators, hardware to disable the power to one or morecomponents, power plug.

In many embodiments, the electronic device is a size constrained,power-sensitive battery operated device with a simple and limited userinterface. Where power is limited, the electronic device might beprogrammed to save power outside of behavior events. For example, aprocessor in the electronic device might be programmed to determine thestart of a behavior event, such as an eating event, and then power upadditional sensors, place certain sensors in a higher performance modeand/or perform additional computations until the processor determines anend of the behavior event, at which point the processor might turn offthe additional sensors, place certain sensors back in a lowerperformance mode and omit the additional computations.

For example, the processor might be programmed to disable allmotion-detection related circuitry, with exception of an accelerometer.The processor could then monitor accelerometer sensor data and if thosedata indicate an actual or prominent food intake activity such as a biteor sip gesture, then the processor could activate additional circuitry,such as a data recording mechanism. The processor might use theaccelerometer sensor data to monitor a pitch of the wearer's arm.

For example, the processor might measure pitch of the wearer's arm untilthe pitch exceeds a certain threshold, perhaps one indicative of a handor arm movement towards the wearers' mouth. Once that is detected, theprocessor can change the state (such as by changing a memory locationset aside for this state from “inactive” or “out-of-event” to “in anaction” or “in-event”) and activate additional circuitry or activate ahigher performance mode of specific circuitry or components. In anotherembodiment, other accelerometer sensor data characteristics such asfirst integral of acceleration (velocity) or the second integral ofacceleration (distance traveled), or characteristics related to orderived from the first and/or second integral of acceleration might beused, as determined from one or more accelerometer axis. A machinelearning process might be used to detect specific movements andtranslate those to gestures.

An end of a food intake event might be detected by the processor byconsidering whether a certain time has expired since a last bite or sipmovement or when other data (meta-data about the wearer,motion-detection sensor data, and/or historical data of the wearer, or acombination of those). Based on those, the processor makes adetermination that a food intake event is not likely and then changesthe state of the electronic device to an inactive monitoring state,possibly a lower power mode.

The lower power mode might be implemented by the processor reducing thesampling rate of the accelerometer and/or gyroscope, powering down thegyroscope, reducing the update rate at which sensor data is transferredfrom the electronic device to a support device, compressing the databefore transferring the data from the sensing electronic device to thesupport device.

In some embodiments of the present disclosure, some of the componentsthat are shown in FIG. 5 as separate components may be combined. As anexample, the processing unit, memory, radio circuitry and PMUfunctionality may entirely or in part be combined in a single wirelessmicrocontroller unit (“MCU”). Other combinations are also possible.Similarly, components that are shown as a single component in FIG. 5 maybe implemented as multiple components. As an example, the processingfunctionality may be distributed across multiple processors. Likewise,data storage functionality may be distributed across multiple memorycomponents. Other examples of distributed implementations are alsopossible.

In another embodiment of the present disclosure, the radio circuitry maynot be present and instead a different interface (such as for example aUSB interface and cable) may be used to transfer data or information toand/or from the electronic device.

Stimulus unit 533 may provide feedback to the user of the electronicdevice. A stimulus unit 533 may include but is not limited to a hapticinterface that applies forces, vibrations or motions to the user, aspeaker or headphones interface that provides sounds to the user, and adisplay that provides visual feedback to the user.

In certain embodiments, the processing and analysis of signals fromsensors embedded in the electronic device can detect when electronicdevice has been disabled, tampered with, removed from the body or is notbeing used. This can be used to conserve power, or to send anotification to the user, a friend or another person who might directlyor indirectly have an interest in being notified if the electronicdevice is not being used properly.

Description Detection/Prediction of Start/End of Food Intake Event

In a preferred embodiment, the electronic device is worn around thewrist, arm or finger and has one or more sensors that generate datanecessary to detect the start and/or end of a food intake event. Theelectronic device may also be integrated in a patch that can be attachedto a person's arm or wrist. The electronic device may also be a moduleor add-on device that can be attached to another device that is wornaround the wrist, arm or finger. Sensors used to detect the start and/orend of a food intake event may among other sensors include one or moreof the sensors described herein.

The raw sensor outputs may be stored locally in memory 529 and processedlocally on processing unit 528 to detect if the start or end of a foodintake event has occurred. Alternatively, one or more sensor outputs maybe sent to the electronic device and/or a central processing and storageunit, either in raw or processed format, for further processing and todetect if the start or end of a food intake event has occurred.Regardless of where the processing for food intake detection occurs,sensor outputs in raw or processed format may be stored inside theelectronic device, inside the electronic device and/or inside thecentral processing and storage unit.

The sensor or sensors that generate data necessary for the detection ofthe start and/or end of a food intake event may be internal to theelectronic device. Alternatively, one or more of the sensors responsiblefor the detection of the start of a food intake event may be external tothe electronic device, but are able to relay relevant information to theelectronic device either directly through direct wireless or wired,communication with the electronic device or indirectly, through anotherdevice. It is also possible that the electronic device and the externalsensor or sensors area able to relay information to the electronicdevice, but are not able to relay information to one another directly.

In case of indirect communication through another device such as amobile phone or other portable or stationary device, such third deviceis able to receive data or information from one or external sensorunits, optionally processes such data or information, and forwardseither the raw or processed data or information to the electronicdevice. The communication to and from the electronic device may be wiredor wireless, or a combination of both.

Examples of sensors that may be external to the electronic device may beone or more sensors embedded in a necklace or pendant worn around theneck, one or more sensors embedded in patches that are attached to adifferent location on the body, one or more sensors embedded in asupplemental second wearable device that is worn around the other arm orwrist or on a finger of the other hand, or one or more sensorsintegrated in a tooth. In some embodiments, the electronic device isworn on one hand or arm but detects movement of the other hand or arm.In some embodiments, electronic devices are worn on each hand.

Information obtained from a non-real-time analysis and learningsubsystem may also be used, optionally in combination with informationfrom one or more sensors 527, to predict or facilitate the detection ofa probable, imminent or actual start/end of a food intake event.

It is often desirable that the detection and/or the prediction of thestart and/or end of a food intake event happens autonomously withoutrequiring user intervention. For example, if the actual, probable orimminent start of a food intake event is predicted or detectedautonomously, this information can be used as a trigger to activate orpower up specific components or circuits that are only needed during afood intake event. This can help conserve power and extend the batterylife of the electronic device. The prediction or detection of an actual,probable or imminent start of a food intake event can also be used toissue a cue or reminder to the user. A cue can for example be sent tothe user to remind him/her to take further actions including, but notlimited to, logging the food intake event or taking a picture of thefood. Upon detection of the start of a food intake event, one or morecues, possibly spread out over the duration of the food intake event, toremind the user that a food intake event is taking place and improvingin-the-moment awareness and/or encourage mindful eating. Cues orreminders may for example be sent through discrete haptic feedback usingone or more stimulus units 533. Other methods using one or more userinterfaces 532, such as for example one or more LEDs, a display message,or an audio signal, are also possible. Alternatively, a mobile devicemay be used to communicate cues, reminders or other information such asfor example portion size recommendations or alternative suggestions toeating to the user.

If the actual, probable or imminent end of a food intake event ispredicted or detected autonomously, this information can be used as atrigger to power down or at least put in a lower power mode one or morecircuits or components of the electronic device that are only neededduring a food intake event. This can help conserve power and extend thebattery life of the electronic device. The detection of the actual,probable or imminent end of a food intake event may also be used tomodify or suspend the feedback provided to the user by one or morestimulus units 533, by one or more of the user interfaces 532, and/or bya mobile device.

In some embodiments of the present disclosure, the detection orprediction of the actual, probable or imminent start and/or end of afood intake event may not be entirely autonomously. For example, theuser may be required to make a specific arm, wrist, hand or fingergesture to signal to the electronic device the actual, probable orimminent start and/or end of a food intake event. The arm, wrist, handor finger gesture is then detected by one or more sensors inside theelectronic device. It is usually desirable that the arm, wrist, hand orfinger gesture or gestures required to indicate the start and/or end ofa food intake event can be performed in a subtle and discrete way. Othermethods may also be used. For example, the user may be asked to push abutton on the electronic device to indicate the start and/or end of afood intake event. Voice activation commands using a built-inmicrophone. Other methods are also possible.

Description of Tracking of Eating Behaviors and Patterns

In a particular embodiment, the electronic device is worn around thewrist, arm or finger and has one or more sensors that generate data thatfacilitate the measurement and analysis of eating behaviors, patternsand habits. Sensors used for measuring and analyzing certain eatingbehaviors and patterns may include one or more of the sensors describedherein.

Relevant metrics that may be used to quantify and track eating behaviorsand eating patterns may include, but are not limited to, the timebetween subsequent bites or sips, the distance between the plate and theuser's mouth, the speed of arm movement towards and/or away from theuser's mouth, and the number of bites or sips during a single foodintake event, derived from the total count of arm movementscorresponding to a bite or sip, specific chewing behavior andcharacteristics, the time between taking a bite and swallowing, amountof chewing prior to swallowing.

In some embodiments, the generation, collection and/or processing ofdata that facilitate the measurement and analysis of eating behaviors,patterns and habits may be continuously, periodically or otherwiseindependently of the start and/or end of a food intake event.Alternatively, the generation, collection and/or processing of data thatfacilitate the measurement and analysis of eating behavior and patternsmay occur only during a food intake event or be otherwise linked to aspecific food intake event. It is also possible that some sensor dataare being generated, collected and/or processed continuously,periodically or otherwise independently of the start and/or end of afood intake event whereas other sensor data are taken during a foodintake event or otherwise linked to a food intake event.

In some embodiments, one camera may be used to capture one or moreimages of food items that are on a plate, table or other stationarysurface, and a second camera may be used to capture one or more imagesof food items that are being held by the user, such as for examplefinger foods or drinks. The use of more than one camera may be desirablein situations where no user intervention is desirable and the position,area of view or focal range of a single camera is not suite to captureall possible meal scenarios.

In one example embodiment, the position, the orientation and the angleof view of the camera are such that an image or video capture ispossible without any user intervention. In such an embodiment, thewearable device may use a variety of techniques to determine the propertiming of the image or video stream capture such that it can capture thefood or a portion of the food being consumed. It may also choose tocapture multiple images or video streams for this purpose. Techniques todetermine the proper timing may include, but are not limited to, thefollowing: sensing of proximity, sensing of acceleration or motion (orabsence thereof), location information. Such sensor information may beused by itself or in combination with pattern recognition or dataanalytics techniques (or a combination of both) to predict the besttiming for the image or video capture. Techniques may include, but arenot limited to, training of a model based on machine learning.

Detailed Description of Specific Embodiments

In one specific embodiment of the present disclosure, the electronicdevice is a wearable device in the form factor of a bracelet orwristband that is worn around the wrist or arm of a user's dominanthand. The electronic device is a mobile phone and central processing andstorage unit is one or more compute servers and data storage that arelocated at a remote location.

One possible implementation of a wearable bracelet or wristband inaccordance with aspects of the present invention is shown in FIG. 6.Wearable device 670 may optionally be implemented using a modulardesign, wherein individual modules include one or more subsets of thecomponents and overall functionality. The user may choose to addspecific modules based on his personal preferences and requirements.

The wearable device 670 may include a processor, a program code memoryand program code (software) stored therein and/or inside the electronicdevice to optionally allow users to customize a subset of thefunctionality of wearable device 670.

Wearable device 670 relies on battery 669 and Power Management Unit(“PMU”) 660 to deliver power at the proper supply voltage levels to allelectronic circuits and components. Power Management Unit 660 may alsoinclude battery-recharging circuitry. Power Management Unit 660 may alsoinclude hardware such as switches that allows power to specificelectronics circuits and components to be cut off when not in use.

When there is no behavior event in progress, most circuitry andcomponents in wearable device 670 are switched off to conserve power.Only circuitry and components that are required to detect or helppredict the start of a behavior event may remain enabled. For example,if no motion is being detected, all sensor circuits but theaccelerometer may be switched off and the accelerometer may be put in alow-power wake-on-motion mode or in another lower power mode thatconsumes less power than its high performance active mode. Theprocessing unit may also be placed into a low-power mode to conservepower. When motion or a certain motion pattern is detected, theaccelerometer and/or processing unit may switch into a higher power modeand additional sensors such as for example the gyroscope and/orproximity sensor may also be enabled. When a potential start of an eventis detected, memory variables for storing event-specific parameters,such as gesture types, gesture duration, etc. can be initialized.

In another example, upon detection of motion, the accelerometer switchesinto a higher power mode, but other sensors remain switched off untilthe data from the accelerometer indicates that the start of a behaviorevent has likely occurred. At that point in time, additional sensorssuch as the gyroscope and the proximity sensor may be enabled.

In another example, when there is no behavior event in progress, boththe accelerometer and gyroscope are enabled but at least one of eitherthe accelerometer or gyroscope is placed in a lower power mode comparedto their regular power mode. For example, the sampling rate may bereduced to conserve power. Similarly, the circuitry required to transferdata from the electronic device to the electronic device may be placedin a lower power mode. For example, radio circuitry 664 could bedisabled completely. Similarly, the circuitry required to transfer hedata from the electronic device to another electronic device may beplaced in a lower power mode. For example, it could be disabledcompletely until a possible or likely start of a behavior event has beendetermined. Alternatively, it may remain enabled but in a low powerstate to maintain the connection between electronic devices but withouttransferring sensor data.

In yet another example, all motion-detection related circuitry,including the accelerometer may be switched off, if based on certainmeta-data it is determined that the occurrence of a particular behaviorevent such as a food intake event is unlikely. This may for example bedesirable to further conserve power. Meta-data used to make thisdetermination may, among other things, include one or more of thefollowing: time of the day, location, ambient light levels, proximitysensing, and detection that wearable device 670 has been removed fromthe wrist or hand, detection that wearable device 670 is being charged.Meta-data may be generated and collected inside wearable device 670.Alternatively, meta-data may be collected inside the mobile phone orinside another device that is external to wearable device 670 and to themobile phone and that can either directly or indirectly exchangeinformation with the mobile phone and/or wearable device 670. It is alsopossible that some of the meta-data are generated and collected insidewearable device 670 whereas other meta-data are generated and collectedin a device that is external to wearable device 670. In case some or allof the meta-data are generated and collected external to wearable device670, wearable device 670 may periodically or from time to time power upits radio circuitry 664 to retrieve meta-data related information fromthe mobile phone or other external device.

In yet another embodiment of the invention, some or all of the sensorsmay be turned on or placed in a higher power mode if certain meta-dataindicates that the occurrence of a particular behavior event, like forexample a food intake event is likely. Meta-data used to make thisdetermination may, among other things, include one or more of thefollowing: time of the day, location, ambient light levels and proximitysensing. Some or all of the meta-data may be collected inside the mobilephone or inside another device that is external to wearable device 670and to the mobile phone and that can either directly or indirectlyexchange information with the mobile phone and/or wearable device 670.In case some or all of the meta-data are generated and collectedexternal to wearable device 670, wearable device 670 may periodically orfrom time to time power up its radio circuitry 664 to retrieve meta-datarelated information from the mobile phone or other external device.

The detection of the start of a behavior event, such as for example afood intake event may be signaled to the user via one of the availableuser interfaces on wearable device 670 or on the mobile phone to whichwearable device 670 is connected. As one example, haptic interface 661inside wearable device 670 may be used for this purpose. Other signalingmethods are also possible.

The detection of the start of a behavior event such as for example afood intake event may trigger some or all of the sensors to be placed orremain in a high-power mode or active mode to track certain aspects of auser's eating behavior for a portion or for the entirety of the foodintake event. One or more sensors may be powered down or placed in alower power mode when or sometime after the actual or probable end ofthe behavior event (the deemed end of the behavior event) has beendetected. Alternatively, it is also possible that one or more sensorsare powered down or placed in a lower power mode after a fixed orprogrammable period of time.

Sensor data used to track certain aspects of a user's behavior, such asfor example a user's eating behavior, may be stored locally insidememory 666 of wearable device 670 and processed locally using processingunit 667 inside wearable device 670. Sensor data may also be transferredto the mobile phone or remote compute server, using radio circuitry 664,for further processing and analysis. It is also possible that some ofthe processing and analysis is done locally inside wearable device 670and other processing and analysis is done on the mobile phone or on aremote compute server.

The detection of the start of a behavior event, such as for example thestart of a food intake event, may trigger the power up and/or activationof additional sensors and circuitry such as for example the cameramodule 651. Power up and/or activation of additional sensors andcircuitry may happen at the same time as the detection of the start of afood intake event or sometime later. Specific sensors and circuitry maybe turned on only at specific times during a food intake event whenneeded and may be switched off otherwise to conserve power.

It is also possible that the camera module only gets powered up oractivated upon explicit user intervention such as for example pushingand holding a button 659. Releasing the button may turn off the cameramodule again to conserve power.

When the camera module 651 is powered up, projecting light source 652may also be enabled to provide visual feedback to the user about thearea that is within view of the camera. Alternatively, projecting lightsource 652 may only be activated sometime after the camera module hasbeen activated. In certain cases, additional conditions may need to bemet before projecting light source 652 gets activated. Such conditionsmay, among other things, include the determination that projecting lightsource 652 is likely aiming in the direction of the object of interest,or the determination that wearable device 652 is not moving excessively.

In one specific implementation, partially depressing button 659 onwearable device 670 may power up the camera module 651 and projectinglight source 652. Further depressing button 659 may trigger cameramodule 651 to take one or more still images or one or more streamingimages. In certain cases, further depressing button 659 may trigger ade-activation, a modified brightness, a modified color, or a modifiedpattern of projected light source 652 either before or coinciding withthe image capture. Release of button 659 may trigger a de-activationand/or power down of projected light source 652 and/or of camera module651.

Images may be tagged with additional information or meta-data such asfor example camera focal information, proximity information fromproximity sensor 656, ambient light levels information from ambientlight sensor 657, timing information etc. Such additional information ormeta-data may be used during the processing and analysis of food intakedata.

Various light patterns are possible and may be formed in various ways.For example, it may include a mirror or mechanism to reflect projectinglight source 652 such that projected light source 652 produces one ormore lines of light, outlines the center or boundaries a specific area,such as a cross, L-shape, circle, rectangle, multiple dots or linesframing the field of view or otherwise giving to the user visualfeedback about the field of view.

One or more Light Emitting Diodes (LEDs) may be used as project lightsource 652. The pattern of visible light may, among other things, beused by the user to adjust the position of the camera, adjust theposition the object of interest and/or remove any objects that areobstructing the line of sight between the object of interest and thecamera.

Projected light source 652 may also be used to communicate otherinformation to the user. As an example, the electronic device me useinputs from one or more proximity sensors, process those inputs todetermine if the camera is within the proper distance range from theobject of interest, and use one or more light sources to communicate tothe user that the camera is within the proper distance range, that theuser needs to increase the distance between camera and the object ofinterest, or that the user needs to reduce the distance between thecamera and the object of interest.

The light source may also be used in combination with an ambient lightsensor to communicate to the user if the ambient light is insufficientor too strong for an adequate quality image capture.

The light source may also be used to communicate information including,but not limited to, a low battery situation or a functional defect.

The light source may also be used to communicate dietary coachinginformation. As an example, the light source might, among other things,indicate if not enough or too much time has expired since the previousfood intake event, or may communicate to the user how he/she is doingagainst specific dietary goals.

Signaling mechanisms to convey specific messages using one or moreprojecting light sources may include, but are not limited to, one ormore of the following: specific light intensities or light intensitypatterns, specific light colors or light color patterns, specificspatial or temporal light patterns. Multiple mechanisms may also becombined to signal one specific message.

Microphone 658 may be used by the user to add specific or custom labelsor messages to a food intake event and/or image. Audio snippets may beprocessed by a voice recognition engine.

In certain embodiments, the accelerometer possibly combined with othersensors may, in addition to tracking at least one parameter that isdirectly related to food intake and/or eating behavior, also be used totrack one or more parameters that are not directly related to foodintake. Such parameters may, among other things, include activity, sleepor stress.

Examples of Using Context to Improve Overall Accuracy

FIG. 7 shows an example of a machine classification system that includesa cross-correlated analytics sub-system 704. Classification output 702may be fed into cross-correlated analytics sub-system 704.Cross-correlated analytics sub-system 704 can make adjustments based oneor more contextual clues to improve the accuracy. In the example ofgesture recognition, an example of a contextual clue could be theproximity in time to other predicted gestures. For example, eatinggestures tend to be grouped together in time as part of an eatingactivity such as a meal or a snack. As one example, cross-correlatedanalytics sub-system 704 could increase the confidence level that apredicted gesture is an eating gesture based on the confidence level anddegree of proximity of nearby predictions.

In another embodiment, cross-correlated analytics sub-system 704 maytake individual predicted gestures 714 from classification output 702 asinputs and may cluster individual predicted gestures into predictedactivities 708. For example, cross-correlated analytics sub-system 704may map multiple bite gestures to an eating activity such as a snack ora meal. Likewise, cross-correlated analytics sub-system 704 could mapmultiple sip gestures to a drinking activity. Other examples of activityprediction based on gesture clustering are also possible.Cross-correlated analytics sub-system 704 may modify the confidencelevel of a predicted gesture based on the temporal spacing and sequenceof predicted activities. As an example, cross-correlated analyticssub-system 704 could decrease the confidence level that a predictedgesture is an eating gesture if it is detected shortly following or amida “brushing teeth” activity. In another example, cross-correlatedanalytics sub-system 704 could decrease the confidence level that apredicted gesture is a drinking gesture if it is detected during orshortly after a brushing teeth activity. In this case, cross-correlatedanalytics sub-system 704 could decide to increase the confidence levelthat the gesture is a rinsing gesture.

Cross-correlated analytics sub-system 704 can adjust a classificationoutput of a predicted gesture based on historical information 712 orother non-gesture meta-data 710 information such as location, date andtime, other biometric inputs, calendar or phone call activityinformation. For example, cross-correlated analytics sub-system 704 mayincrease the confidence level that a predicted gesture is an eatinggesture or a predicted activity is an eating activity if GPS coordinatesindicate that the person is at a restaurant. In another example,cross-correlated analytics sub-system 704 may increase the confidencelevel that a predicted gesture is an eating gesture or a predictedactivity is an eating activity if it occurs at a time of day for whichpast behavior indicates that the user typically engages in eating atthis time of the day. In yet another example of the present disclosure,cross-correlated analytics sub-system 704 may increase the confidencelevel that a predicted gesture is an eating gesture or that a predictedactivity is an eating activity if the predicted gesture or predictedactivity is preceding or following a calendar event or phone callconversation if past behavior indicates that the user typically eatspreceding or following similar calendar events (e.g., with sameattendee(s), at certain location, with certain meeting agenda, etc.) orphone call conversation (e.g., from specific phone number). While theabove examples reference eating, it will be apparent to one skilled inthe art that this could also be applied to gestures other than eating.In the general case, the machine classifier with cross-correlatedanalytics sub-system uses contextual clues, historical information andinsights from proximity sensing in time to improve accuracy, where thespecific contextual clues, historical information and insights fromproximity sensing in time and how they are applied is determined bymethods disclosed or suggested herein.

In some embodiments of the present disclosure, Classification Output 702may include additional features or gesture time envelope information.Cross-correlated analytics sub-system 704 may process such additionalfeatures or gesture time envelope information to determine or extractadditional characteristics of the gesture or activity. As an example, inone embodiment of the present disclosure, cross-correlated analyticssub-system 704 derives the estimated duration of the drinking gesturefrom the gesture time envelope and this information can be used bycross-correlated analytics sub-system 704 or by one or more systems thatare external to the machine classifier system to estimate the fluidintake associated with the drinking gesture.

In another embodiment, cross-correlated analytics sub-system 704 mayderive the estimated duration of an eating gesture from the gesture timeenvelope and this information may be used by the cross-correlatedanalytics sub-system 704 or by one or more systems that are external tothe machine classifier system to estimate the size of the biteassociated with the eating gesture. Cross-correlated analyticssub-system 704 may combine the predicted drinking gestures with othersensor data to predict more accurately if someone is consuming a drinkthat contains alcohol and estimate the amount of alcohol consumed.Examples of other sensor data may include but are not limited tomeasuring hand vibration, heart rate, voice analysis, skin temperature,measuring blood, breath chemistry or body chemistry.

Detector sub-system 700 may predict a specific eating or drinking methodand cross-correlated analytics sub-system 704 may combine theinformation obtained from detector sub-system 700 about specifics of theeating or drinking method with additional meta-data to estimate thecontent, the healthiness or the caloric intake of the food. Examples ofeating/drinking methods may include but are not limited to eating withfork, eating with knife, eating with spoon, eating with fingers,drinking from glass, drinking from cup, drinking from straw, etc.).Examples of meta-data may include but are not limited to time of day,location, environmental or social factors.

Medication Dispensing System

FIG. 8 shows a high level functional diagram of a medication dispensingsystem covered by the current disclosure. A medication dispensing systemmay in part include one or more of the following: a dietary tracking andfeedback system 1902, one or more sensor units 1900, a measurementsensor processing unit 1909, a medication dosing unit 1906, one or moremeasurement sensor units 1904 and a medication dispensing unit 1908.

In one embodiment of the current disclosure, the medication to beadministered is insulin, the measurement sensor unit 1904 is acontinuous glucose monitor sensor that measures interstitual glucoselevels, medication dispensing unit 1908 is an insulin pump andmedication dosing unit 1906 is the insulin dosage computation unit of anautomated insulin delivery system (a.k.a., artificial pancreas).

Each of the elements shown in FIG. 8 might be implemented by suitablestructure. For example, these could be individual hardware elements orimplemented as software structures in a wearable device, an ancillarydevice that communicates with a wearable device, or a server coupledover a network to the ancillary device and/or a wearable device. Someelements might be entirely software and coupled to other elements thatare able to send and/or receive messages to or from the processorexecuting the software. For example, medication dispensing unit 1908might be an embedded hardware system that injects microdoses ofmedication in response to instructions given by a software system andthus would only require minimal processing on-board.

Dietary tracking and feedback system 1902 maybe implemented as describedelsewhere herein and may monitor the outputs of one or more sensorunit(s) to determine the actual, probable or imminent start of a foodintake event. Upon detection of an actual, probably or imminent start ofa food intake event, or some time thereafter, it may send a signal 1903to medication dosing unit 1906 to inform medication dosing unit 1906that an actual, probable or imminent start of a food intake event hasbeen detected. Medication dosing unit 1906 may use this information tochange its state to a “meal-in-progress” state.

Upon entering the meal-in-progress state or some time thereafter,medication dosing unit 1906 may calculate an initial meal medicationdosage to be administered and send one or more messages to medicationdispensing unit 1908. Alternatively, the medication dosage to beadministered in connection with the start of a food intake event mayhave been pre-configured or calculated ahead of the occurrence of thefood intake event. Upon receiving those messages 1907, medicationdispensing unit 1908 may initiate delivery of the medication.

The medication dispensing unit may deliver the medication all at ones oraccording to a delivery schedule. The delivery schedule may bedetermined and communicated to the insulin delivery system by themedication dosing unit. The delivery schedule may be determined uponentering the meal-in-progress state or some time thereafter. Thedelivery schedule may also be pre-configured ahead of the occurrence ofthe food intake event.

The initial meal medication dosage and/or delivery schedule may coverthe entire anticipated medication dosage for the food intake event.Alternatively, the initial meal medication dosage and/or deliveryschedule may only cover a portion of the entire anticipated medicationdosage for the food intake event, and additional medication dosages areexpected at a later time during or after the food intake event.

Medication dosing unit 1906 may take into account additional inputs whencalculating an initial medication dosage and or initial deliveryschedule. Some inputs may be related to current or recent measurements,current or recent user activities and behaviors or other informationcorresponding to a current or recent state or condition of a user. Otherinputs may be related to historical measurements, historical useractivities and behaviors or other information corresponding to a paststate or condition of a user.

Examples of Additional Inputs

Medication dosing unit 1906 may take into account one or more outputs1910 from measurement sensor processing unit 1909. Medication dosingunit 1906 may perform additional processing steps on outputs 1910. Forexample, measurement sensor unit 1904 may be a continuous glucosemonitor (“CGM”), and outputs 1910 of measurement sensor processing unit1909 may be interstitual glucose level readings. Outputs 1910 may forexample be updated every couple of minutes. Other update frequencies arealso possible. Outputs 1910 may also be updated continuously. Medicationdosing unit 1906 may take into account one or more interstitual glucoselevel readings. For example, medication dosing unit 1906 may take intoaccount the most recent reading. Medication dosing unit 1906 maycalculate certain parameters that are indicative of changes ininterstitual glucose level readings. For example, medication dosing unit1906 may calculate the minimum, mean, maximum, standard deviation, slopeor second derivative of interstitual glucose level readings over one ormore time windows. A time window may span a period of time preceding thetransition to the meal-in-progress state, span a period of time thatincludes the transition to the meal-in-progress state, or span a periodof time some time after the transition to the meal-in-progress state.Other or additional measurement sensor units such as heart rate, bloodpressure, body temperature, hydration level, fatigue level are alsopossible. An insulin delivery system may also take into account a user'scurrent location.

Medication dosing unit 1906 may also take into account other inputs suchas information related to a user's current or recent physical activity,sleep, stress etc. Medication dosing unit 1906 may also take intoaccount personal information such as gender, age, height, weight, etc.

Medication dosing unit 1906 may also take into account informationrelated to a user's medication dosing needs, such as for example auser's insulin basal rates, a user's insulin-to-carb ratios, and auser's insulin correction factor. This information may be entered orconfigured by the user, by a caregiver or by a health record orhealthcare maintenance system. Information related to a user'smedication dosing needs may also be derived from historical datacollected and stored by the medication dispensing system. For example,the dosage of medication (e.g. insulin) delivered by the medicationdispensing unit in a period of time preceding the current food intakeevent. Medication dosing unit 1906 may take into account medicationdosages delivered in connection with one or more prior food intakeevents that occurred in the past at or around (e.g., within a specifiedtime window) the same time of day, and/or the same day of the week.

Medication dosing unit 1906 may also take into account the medicationstill active from previous medication dispensing events, such as forexample the insulin-on-board.

Medication dosing unit 1906 may also include parameters related to foodintake events that occurred in the past at or around (e.g. within aspecified time window) the same time of day, and/or the same day of theweek, and/or at the same location. Medication dosing system 1906 may forexample look at the duration of past food intake events, the estimatedamounts consumed during past food intake events, the average pace ofeating during past food intake events, the eating methods of past foodintake events, the type of utensils or containers used during past foodintake events, or the amounts of carbohydrates consumed during past foodintake events. Other parameters are also possible. Some of theseadditional parameters (e.g., duration or pace) may be computed by thefood intake tracking and feedback system without requiring any userintervention. In other cases, a user intervention, input or confirmationby the user may be necessary.

Medication Dispensing Schedule

Medication dosing unit 1906 may instruct medication dispensing unit toadminister the initial medication dosage all at once, or may specify adelivery schedule for administering the medication. In one embodiment ofthe current disclosure, medication dosing unit 1906 computes themedication dosage as well as a schedule for the delivery of themedication. As an example, medication dosing unit 1906 may determinethat 5 units of insulin need to be delivered and may specify a deliveryschedule as follows: 2 units to be administered immediately, 1 unitafter 2 minutes, 1 unit after 5 minutes and 1 unit after 7 minutes. Thisis just one example and other time profile structures are of coursepossible.

Medication dosing unit 1906 may communicate both the medication dosageand schedule to medication dispensing unit 1908. Alternatively,medication dosing unit 1906 may manage the schedule and send one or moremessages to medication dispensing unit 1908 every time medication needsto be administered along with the dosage of medication to beadministered.

In a preferred embodiment of the current disclosure, medication dosingunit 1906 may upon entering the meal-in-progress state or some timethereafter instruct 1907 medication dispensing unit 1908 to initiatedelivery of medication. It may for example instruct medicationdispensing unit 1908 to deliver one or more small micro-doses ofmedication.

Additional Dosages and Dosage Adjustments

During the food intake event and/or some time after the food intakeevent, medication dosing unit 1906 may periodically (e.g., every coupleof minutes) or continuously monitor in part one or more inputs 1905 frommeasurement sensor processing unit(s) 1909, and/or one or more inputs1903 from dietary tracking and feedback system 1902 to determine if andhow much additional medication should be administered or whether ascheduled medication dispensing should be adjusted. Other inputs such asinputs described in earlier sections of this disclosure may also betaken into account.

When computing a medication dosage or medication dosage adjustment,medication dosing unit 1906 may take into account whether or not a foodintake event is in progress. If a food intake event is not or no longerin progress, medication dosing unit 1906 may for example take intoaccount the time since the end of the last food intake event. If a foodintake event is in progress, medication dosing unit 1906 may for exampletake into account the time elapsed since the start of the current foodintake event, the average or median pace of eating since the start ofthe current food intake event, the estimated total amounts consumedsince the start of the current food intake event. Other examples arealso possible.

Medication dosing unit 1906 may also take into account one or morerecent inputs from measurement sensor processing unit 1909. For example,in case medication dosing unit 1906 is an insulin dosing unit in anautomated insulin delivery system (aka artificial pancreas) andmeasurement sensor unit 1909 is a CGM, medication dosing unit 1906 maytake into account the value of the most recent interstitual glucoselevel reading and/or the change in interstitual glucose level readingsover a time window immediately or closely preceding the current time. Ifthe most recent interstitual glucose level reading is below a specifiedthreshold and/or a change in interstitual glucose level readings isexceeding a specified negative threshold, medication dosing unit 1906may determine to adjust the insulin dosage down, or suspend insulindelivery until the interstitual glucose level readings reaches a secondspecified threshold and/or the change in interstitual glucose levelreadings is no longer exceeding a second specified negative threshold,has turned positive or is exceeding a specified positive threshold.

In some embodiments of the current disclosure, upon detecting of aconcerning measurement sensor processing unit output, medication dosingunit 1906 may send an alert to the user, one or more of his caregivers,a healthcare provider, a monitoring system or to an emergency responsesystem, or to a third party who may have a direct or indirect interestin being informed about the occurrence of such episodes.

Likewise, if the most recent interstitial glucose level reading exceedsa specific threshold and/or a change in interstitual glucose levelreadings is exceeding a specified positive threshold, medication dosingunit 1906 may determine that an additional medication dosage should beadministered, that an already scheduled medication dosage needs to beadjusted to a larger dosage or delivered at a time earlier than thecurrently scheduled time. Medication dosing unit 1906 may optionallytake into account additional inputs from dietary tracking and feedbacksystem 1902 to compute the additional medication dosage or medicationdosage adjustment.

Medication dosing unit 1906 may send one or more messages 1907 tomedication dispensing unit 1908 to inform medication dispensing unit1908 of the additional or adjusted medication dispensing requirements.

Upon detection of an actual or imminent end of a food intake event, orsome time thereafter, dietary tracking and feedback system 1902 may senda signal 1903 to medication dosing unit 1906 to inform medication dosingunit 1906 that an actual or imminent end of a food intake event has beendetected. Medication dosing unit 1906 may use this information to changeits state to a “no-meal-in-progress” state. In specific embodiments, themedication dosing unit, when in a no-meal-in-progress state mayperiodically or continuously monitor in part one or more inputs 1905from measurement sensor processing unit(s) 1909, and/or one or moreinputs 1903 from dietary tracking and feedback system 1902 to determineif and how much additional medication should be administered or whethera scheduled medication dispensing should be adjusted. Other inputs suchas inputs described in earlier sections of this disclosure may also betaken into account. When in the no-meal-in-progress state, the frequencyof monitoring and/or updating/adjusting medication dosing may differentfrom the frequency of monitoring and/or updating/adjusting medicationdosing when in the “meal-in-progress” state. The algorithms used todetermine a medication dosage or medication dosage adjustment may alsobe different between the “meal-in-progress” state and the“no-meal-in-progress” state.

Description of Medication Dosing Learning System

In some embodiments of the current disclosure, medication dosing unit1906 may collect, and store data and information about food intakeevents. In some embodiments, medication dosing unit 1906 may performadditional processing steps on collected data and information beforestoring. Processing steps may be filtering, averaging, applyingarithmetical operations, and applying statistical operations. Otherprocessing steps are also possible.

Data and information about food intake events might be stored as dataelements in a database that maintains data records about food intakeevents.

Data elements may be event data elements and contain information orparameters that characterize the food intake event. Such information mayinclude, but is not limited to, the event start time, the day of weekthe event occurred, the date of the event, the duration of the event,the event end time, metrics associated with the speed or pace at whichsubject is eating or drinking, metrics associated with the amounts offood or liquid consumed during the event.

Data elements may also be measurement data elements and containinformation or parameters that characterize one or more signals measuredby one or more measurement sensor units 1904 and processed by one ormore measurement sensor processing units 1909. Such information mayinclude, but is not limited to, sensor reading levels corresponding tospecific times in connection with the food intake event, or the average,minimum, or maximum sensor reading levels corresponding to specific timewindows in connection with the food intake event. Specific times mightfor example be the start of the food intake event, periodic orpre-defined points in time during the food intake event, the end of thefood intake event, or periodic or pre-defined times after the foodintake event. Other times are also possible. Specific time windows mightfor example be a duration of time immediately preceding the start of thefood intake event, a duration of time prior to the start of the foodintake event, the duration of the food intake event, a specific durationof time within the food intake event, a duration of time immediatelyfollowing the end of a food intake event or a duration of time some timeafter the end of a food intake event.

In one specific embodiment the medication dosing unit is an automatedinsulin delivery system, and the sensor reading levels are interstitualglucose reading levels obtained from a continuous glucose monitoringsensor.

Data elements may also be dosage data elements and contain informationor parameters that characterize the medication dosage and deliveryschedule in connection with the food intake event.

Other data elements are also possible. One or more event data elements,one or more measurement data elements and/or one or more dosage dataelements of the same food intake event may be recorded as a singlerecord entry in a database. Event data elements, measurement dataelements and/or dosage data elements may also be recorded as separaterecords in a database. Other data structures consistent with theteachings herein might also be used, or used instead.

Medication dosing unit 1906 may include a processing and analysissubsystem.

The processing and analysis subsystem may use statistics, machinelearning or artificial intelligence techniques on entries in thedatabase to build a model that recommends an adequate medication dosageand/or delivery schedule. The processing and analysis subsystem may beused to recommend an initial medication dosage, and/or to recommendadditional medication dosage(s) or dosage adjustment(s).

FIG. 9 is an illustrative example of a machine learning system thatmight be used with other elements described in this disclosure. Themachine learning system of FIG. 9 includes a dosage training subsystem2020 and a dosage predictor subsystem 2021. In some embodiments of thepresent disclosure, the machine learning system may include additionalsubsystems or modified versions of the subsystems shown in FIG. 2.Dosage training subsystem 2020 might use event data elements 2022,measurement data elements 2023 and dosage data elements 2024 as inputs.The dosage training sub-system applies machine learning techniques tobuild a model that recommends an adequate medication dosage and/or amedication dispensing schedule. It might use supervised learningtechniques on one or more entries from the database to train the model.Event data element(s) 2022 and/or measurement data element(s) 2023 mightbe used as features for the model. One or more dosage data elements 2024might be used as labels. Trained model(s) 2025 and/or 2029 is then usedin dosage predictor subsystem 2021 to generate a medication dosagerecommendation and/or medication dispensing recommendation correspondingto a new unlabeled data input 2026.

Medication dosing unit 1906 may include a processing unit and performadditional processing on the data elements and analyze the data toextract information about the user's eating and drinking activities andbehaviors, sensor measurements (e.g., glycemic control) and/ormedication regimen. Processing may include but is not limited tofiltering, extracting specific data elements, modifying data or dataelements, combining data or data elements. Analysis may also includecomparing specific data elements to data that is stored in a look uptable or database, correlating data elements to data elements that wereobtained at an earlier time and/or from different subjects. Medicationdosing unit 1906 may store raw or processed data in one or more datastorage unit(s). Storage may be temporary or permanent.

In some variations, records in the database may be subdivided intogroups (e.g., based on meal type—breakfast, lunch, dinner, snack) and adifferent model may be used for each subgroup. Alternatively, the samemodel may be used but it may be trained using data from only one or aselective set of subgroups. In other variations, instead of a supervisedmachine learning approach (i.e., where features are specified manually),unsupervised learning may be used instead. With unsupervised learning,the classifier will autonomously generate features from the raw datasetprovided.

Medication dosing unit 1906 may collect, and store data and informationabout other user activities. For example, medication dosing unit 1906may collect information or data about a user's physical activity, sleepactivity, sexual activity. Medication dosing unit 1906 may also collectand store information about a user's stress, heart rate, blood pressureetc. In some embodiments, medication dosing unit 1906 may performadditional processing steps on collected data and information beforestoring. Processing steps may be filtering, averaging, applyingarithmetical operations, and applying statistical operations. Otherprocessing steps are also possible. Medication dosing unit 1906 may linkthis data and information to one or more food intake events. Data andinformation about food intake events might be stored as data elements ina database that maintains data records about food intake events. Thesedata elements may also be used as inputs to processing and analysissubsystem of medication dosing unit 1906. For example, these dataelements may be used as additional or alternative event data inputs todosage training subsystem 1920. These data elements may for example befeatures for the model.

Medication dosing system 1906 may also collect inputs such asinformation related to a user's physical activity, sleep, stress etc.Medication dosing system 1906 may for example compare a user's currentor recent physical activity to past physical activity and use the outputof that comparison into the calculation of a medication dosage.

While not illustrated in detail in FIG. 9, the machine learning system,dosage training subsystem 2020, and dosage predictor subsystem 2021 canbe implemented using various structural elements. For example, dosageprediction might be implemented using computer hardware, such as aprocessor, program code memory and program code stored in the programcode memory. This might be a separate embedded unit, or might beimplemented on a processor with memory that is used for other functionsand tasks as well as dosage prediction. This processor and memory mightbe built into a wearable device, a mobile device that communicates witha wearable device, a server that is in communication, directly orindirectly, with a wearable device or sensors, or some combination ofthe above. Other elements might similarly be implemented, such asstorage for event data elements 2022, storage for measurement dataelements 2023 and storage for dosage data elements 2024, program codethat implements the machine learning techniques to build a model thatrecommends an adequate medication dosage and/or a medication dispensingschedule, storage for the model, storage for a database to train themodel, and hardware circuitry needed to communicate messages, such assending a medication dosage recommendation message, a medicationdispensing recommendation message, or a signal to a hardware device thatdispenses medication.

In certain embodiments, the medication dispensing system of FIG. 8 mayoperate without any manual intervention or at least not requiring anymanual intervention. In other embodiments, the medication dispensingsystem of FIG. 8 may require some manual intervention. In one example,medication dosing unit 1906 may calculate a medication dosage and/ormedication delivery schedule but, instead of instructing the medicationdispensing unit 1908 to initiate or schedule delivery of medication, itmay send a message to the patient, one or more caregivers of thepatient, a healthcare professional, a monitoring system, etc. to confirmthe proposed medication dosage and/or medication delivery schedule. Themessage can be a text message, a push notification, a voice message,etc., but other message formats are also possible.

In some embodiments, the patient, caregiver, etc. may have an option toalter the proposed medication dosage and/or medication deliveryschedule. Upon receiving a confirmation from the patient, caregiver,etc., medication dosing unit 1906 may send one or more instructions tothe medication dispensing unit 1908 to initiate or schedule delivery ofmedication. The instruction(s) may also be sent by a device or unitother than the medication dosing unit 1906. As an example, theinstruction(s) may be sent to the medication dispensing unit 1908directly from the device on which the patient, caregiver etc. receivedthe message to confirm the medication dosage and/or medication deliveryschedule. Other user interventions are also possible, such as allowingfor a “snooze” function to move a message to a predetermined time in thefuture.

Interpretation

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments ofthe invention and does not pose a limitation on the scope of theinvention unless otherwise claimed. No language in the specificationshould be construed as indicating any non-claimed element as essentialto the practice of the invention.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and it should be understood that combinations, additions,re-arrangements, and the like are contemplated in alternativeembodiments of the present invention. Thus, while the invention has beendescribed with respect to exemplary embodiments, one skilled in the artwill recognize that numerous modifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. An event detection system that includes sensors to detect movement and/or other physical inputs related to a user, which the event detection system can process to identify gestures of the user, a method of further processing comprising: determining a type of an inferred event, wherein the inferred event is an actual event occurring with or about the user or is, based on sensor inputs, deemed by the event detection system to be an event; determining an external trigger time, based on one or more of data related to the inferred event, the type of the inferred event, and/or a timing of the inferred event, wherein the external trigger time is a time determined relative to an anchor time of the inferred event; and according to the external trigger time, initiating a computer-based action in response to the inferred event.
 2. The event detection system of claim 1, wherein the sensors include sensors external to a main event detection unit.
 3. The event detection system of claim 1, wherein the computer-based action in response to the inferred event comprises sending a trigger signal to an ancillary system that is operable independent of the event detection system and independent of the trigger signal.
 4. The event detection system of claim 1, wherein the event detection system is configured to determine, using historical data, machine learning, rule sets, and/or other techniques for processing data, an inferred event related to the user sensed by the sensors.
 5. The method of claim 1, wherein the type of the inferred event is at least one of a food intake event, a drinking event, a smoking event, a personal hygiene event, and/or a medication related event.
 6. The method of claim 1, wherein the external trigger time is determined from when the inferred event is inferred to have started, when the inferred event is inferred to be ongoing, and/or when the inferred event is inferred to have concluded.
 7. The method of claim 1, wherein the computer-based action in response to the inferred event is one or more of (1) obtaining other information to be stored in memory in association with data representing the inferred event, (2) interacting with the user to provide information, coaching advice or a reminder, (3) interacting with the user to prompt for user input, (4) sending a message to a remote computer system, (5) sending one or more inputs to a medication dispensing system, and/or (6) sending a message to another person.
 8. The method of claim 7, wherein the computer-based action comprises inputs to the medication dispensing system and wherein the medication dispensing system is an automated, or partially automated, insulin delivery system.
 9. The method of claim 7, wherein the medication dispensing system comprises an insulin dosage calculator.
 10. The method of claim 9, wherein the insulin dosage calculator computes information from an object information retrieval subsystem about what is being consumed.
 11. The method of claim 1, further comprising updating one or more of an inventory database, a medication log, an inventory, and/or a production line database in response to the inferred event.
 12. The method of claim 1, wherein the computer-based action comprises an action to retrieve information about at least one object the user is interacting with.
 13. The method of claim 12, wherein the retrieved information is information retrieved over a wireless link.
 14. The method of claim 13, wherein the wireless link uses one of (1) near-field-communication technology, (2) Bluetooth technology, (3) Bluetooth Low Energy technology, (4) a derivative of Bluetooth technology that is at least partially consistent with Bluetooth technology, or (5) a derivative of Bluetooth Low Energy technology that is at least partially consistent with Bluetooth Low Energy technology. A method of managing a food log based in part on sensor data from sensors on a device worn by a user for logging the user's food intake is also provided, that might include identifying a current eating event; further identifying one or more characteristics of the current event, from the current eating event and the characteristics of the current event, and from historical data of past eating events, using a training system to identify additional characteristics of a current eating event; and periodically uploading data representing characteristics of eating events to a food log server.
 15. A method of managing a food log based in part on sensor data from sensors on a device worn by a user for logging the user's food intake, the method comprising: identifying a current eating event; identifying one or more characteristics of the current eating event; from the current eating event and the characteristics of the current event, and from historical data of past eating events, using a training system to identify additional characteristics of a current eating event; and periodically uploading data representing characteristics of eating events to a food log server.
 16. A method of managing a food log based in part on sensor data from sensors on a device worn by a user for logging the user's food intake, the method comprising: identifying a current eating event; identifying one or more characteristics of the current eating event; periodically uploading data representing characteristics of eating events to a food log server; comparing uploaded characteristics of food intake events to second set of eating characteristics that have been obtained through a different mechanism; and as a result of comparing, take one or more of the following actions (1) merge characteristics from both datasets correspond to the same food intake event into a single eating event data record of a food logging system, and (2) adding entries to the food log for the events absent from the food log but represented by identified characteristics of the current eating event.
 17. The method of claim 15, further comprising: using historical data of past eating events to identify additional characteristics of a current eating event; and uploading additional characteristics to a food log server.
 18. The method of claim 15, wherein periodically uploading data comprises periodically uploading data at conclusions of detected eating events.
 19. The method of claim 15, wherein periodically uploading data comprises uploading data at times that correspond to anchor times of detected eating events.
 20. The method of claim 15, wherein the data representing characteristics of eating events is derived from inferences derived from sensed user movements during eating events.
 21. The method of claim 15, wherein the data representing characteristics of eating events is partially inferred from sensed user movements during eating events and meal signatures are added to the food log so as to allow the user to edit the food log while being reminded of the meal signature.
 22. The method of claim 15, wherein recording the inferred event in the food log comprising recording one or more of information inferred by the event detection system and information retrieved from one or more objects the user interacted with as part of the inferred event, wherein user interaction is detected using sensors on a device worn by a user and sensor signals from those sensors or from devices designed to read electronic tags on the user or the food.
 23. The method of claim 15, wherein the food log comprises information about eating or drinking events including one or more of a time of an event, a duration of the event, a location of the event, metrics related to pace of consumption during the event, metrics related to quantities consumed during the event, eating methods, and/or utensils used.
 24. The method of claim 15, wherein the food log comprises information about items being eaten or drunk, wherein such information is obtained from electronically readable tags attached to the items, the method further comprising sending a tag reader trigger signal to a tag reader to read the electronically readable tags when an inferred event is detected.
 25. The method of claim 24, further comprising prompting the user to reposition the items and the tag reader to a relative position in which the tag reader can read the electronically readable tags attached to the items.
 26. A method of sending signals or messages in response to an event timed relative to the event, the method comprising: identifying behavior indicators; from the behavior indicators and timestamps of the behavior indicators, and from historical data of past behavior events and behavior indicators, using a training system to predict likely future events including a time of such likely future events; and reading a rule set; when the rule set indicates that a particular signal should be sent when a particular event is anticipated and when the particular event is indeed anticipated with at least a predefined confidence level, outputting the particular signal to an ancillary system; and when the rule set indicates that a particular message should be sent when the particular event is anticipated and when the particular event is indeed anticipated with at least the predefined confidence level, sending the particular message.
 27. The method of claim 26, wherein identifying behavior indicators comprises: detecting at least one gesture of a user wearing a wearable device having a plurality of sensors to detect movement and other physical inputs related to a user; receiving sensor inputs from the plurality of sensors; reading data from external data sources; and from the sensors inputs and the data, identifying the behavior indicators.
 28. The method of claim 26, wherein the particular event is an eating event, wherein the particular signal is a signal to trigger an insulin microdosing system and the particular message is a message to the user of a wearable device indicating that the start of the eating event has been detected
 29. The method of claim 26, wherein the particular event is such that it could not be predicted from the sensor inputs alone or the data from external sources alone.
 30. The method of claim 26, wherein the particular event is an eating event and behavior indicators correlate with triggers, wherein triggers include one or more of a sleep pattern, a stress level, an activity level, the user's location, people surrounding the person, vital signs, a hydration level, a fatigue level, or a heart rate.
 31. The method of claim 26, wherein the signal or message is a function of a level of user engagement.
 32. The method of claim 31, further comprising assessing the level of user engagement at a given time or over a time window, combined with user responses to certain messages. 